Data Scientist Scarcity: Automation Is the Answer

In a meeting just last week, one Fortune 100 CTO referenced a familiar Harvard Business Review article that named the Data Scientist as the 21st century’s most attractive job, a bold comment based on the scarcity of qualified individuals. I thought it might be old news by now, but was surprised to find this article from last October still fresh in the minds of today’s executives.

In a meeting just last week, one Fortune 100 CTO referenced a familiar Harvard Business Review article that named the Data Scientist as the 21st century’s most attractive job, a bold comment based on the scarcity of qualified individuals. I thought it might be old news by now, but was surprised to find this article from last October still fresh in the minds of today’s executives.

Why should we care, though? While scarcity tends to make things more desirable, the century is young, and history will likely be less kind to the data scientist than the Review predicts. What’s more likely to happen as the decades progress is collaboration between Joe Data Scientist and powerful algorithms to solve big data challenges with automation.

From Agriculture to Algorithms

Just as tractors symbolized the dawn of a revolution in farming and signaled that the old way of farm life was coming to an end, automated data analysis will change our relationship with data. Think of it this way: Data is like dirt. It’s everywhere and it’s plentiful. Working the land used to be incredibly manually intensive. We were an agrarian society because we had to be. Automation changed all that. Similarly, data analysis is currently incredibly manually intensive. Automation will change all that. The key to any process—and the signal that a revolution is at hand—is when the burden is moved from man to machine.

At the moment, the data scientist represents a stage in the evolution of Big Data analysis; a stopgap until the technology emerges that will do the job for him. The progress of what algorithms can do is far outpacing the mortal speed of data scientists, and businesses are taking note of how much time, energy, and lost profit can be saved by alleviating the pressure on the data middleman.

Companies that insist that the paradigm has not yet switched can’t afford the price of being left behind. Data algorithms are this century’s solution, and here’s why.

Queried Out

Organizations realize the value in harnessing information – that’s why we collect it, store it, process and analyze it. Query-based searches utilized by business analysts and data scientists often miss the hidden value in their collected data because they have to know the right questions to ask their data.  Missed insight compounds into missed opportunities and profits.

Human directed queries do not allow for unimagined possibilities, un-thought of connections. What’s needed is a way to automate the discovery process. It’s not a matter of asking more questions, faster. It’s a matter of finding the right questions. And the way to do that is through algorithmic analysis.

Algorithms: This Century’s Tractors

Think again of the dirt and tractor analogy. Just as farming vast acreage requires automaton, analyzing vast volumes of data requires automation. Algorithms are this century’s tractors and our potential harvest is more valuable information. The cutting edge of Big Data analytics is less about brute human force and more about mathematical finesse.

Algorithms will replace human queries and will allow analysts to stop spending time plowing through fields of data and instead spend more time figuring out what to do with the insights they’ve harvested. With the help of algorithms, data scientists and business analyst will become even more valuable and productive as the Big Data Century unfolds.