The Future of Data Science

The last couple of weeks were great for the future of data science. First Wikibon, and then IDC, promised a big data market in 2015 of between $16.9 billion (IDC) to $32.1 billion (Wikibon) (more on these reports in Chuck Hollis’ Big Data: From Meme to Marketplace).

The last couple of weeks were great for the future of data science. First Wikibon, and then IDC, promised a big data market in 2015 of between $16.9 billion (IDC) to $32.1 billion (Wikibon) (more on these reports in Chuck Hollis’ Big Data: From Meme to Marketplace). And the Strata conference showcased the promising startups and data scientists that are going to make the big big data market a reality (see Daniel Tunkelang’s excellent summary here).  Tim O’Reilly aptly summarized all of this excitement by declaring that “data science is the new black.”

So where do we go from here? How will data scientists’ careers shape up over the next decade?

We all know that predictions are difficult to make, especially about the future. But here are three possible scenarios, focused on the future consumption of big data analytics by established enterprises, for where data science will be in 2020:

Big Data Ventures

Data Science will be practiced exclusively by companies specializing in big data analytics. These small ad-hoc or VC-funded teams of data scientists, comprised of data cleaning, analysis, and visualization experts, will either provide a generic service for any big data problem or a domain-specific service. In both cases the analysis will be done mostly on the basis of the growing stores of public data with only the occasional integration with an organization’s proprietary data. Large and small enterprises will be slow to adopt these services and will continue to rely on legacy BI tools for their limited data mining needs.  Data scientists will be a special breed, the only people with the experience and expertise to wrestle with the messy explosion of both digital (and dirty) data and big data tools.

Big Data Accountants   

Data Science will become a specialized, in-house function, similar to today’s Accounting, Legal, and IT departments. Leading universities will establish stand-alone Data Science departments, conferring data science degrees, Bachelor’s to Ph.D. Big data startups will be absorbed by large IT vendors who will develop big data “enterprise applications” and will convince corporate buyers that big data analytics is the next big thing. Data scientists will be either academics, independent consultants, or members of the corporate data science function, where they will rise to the title of CDO (understood in leading organizations as Chief Decision Officer and by laggards as Chief Data Officer).

Everybody’s a Big Data Expert

The vision of “data democracy” will come true and everybody in the organization will create and consume big data. Data science fundamentals will be thoroughly integrated in all levels of management education.  Mobile, easy-to-use big data analytics tools will follow the adoption curve of Excel and PowerPoint, in a market dominated by one or two large IT vendors. This dominant (tablet-based?) big data “platform” (or platforms) will also be heavily used by consumers at home to make sense of their personal big data.  Data science will not be a specific discipline or job category.

The above scenarios are of course intentionally extreme and by 2020 we may see elements from each scenario combined into a new reality. While all of these elements are based on current trends, the question is: which will survive? And what probability would you assign to each scenario?