Who will manage Big Data?

An interesting McKinsey Global Institute (MGI) analysis looks at the vast amount of enterprise information these days–and the challenges that organizations will face in trying to manage it. The report, Big data: The next frontier for innovation, competition, and productivity, explores things such as the state of digital data and how different domains can use large data sets to create value. Here’s an excerpt:

MGI’s analysis shows that companies and policy makers must tackle significant hurdles to fully capture big data’s potential. The United States alone faces a shortage of 140,000 to 190,000 people with analytical and managerial expertise and 1.5 million managers and analysts with the skills to understand and make decisions based on the study of big data (exhibit). Companies and policy makers must also tackle misaligned incentives around issues such as privacy and security, access to data, and technology deployment.

To read the entire report, click here (registration required).

Reactions

While the sheer numbers surprised me a little, the issues the report raised certainly did not. Industry observers have long noted the shortage of skilled people who can handle the increasing amounts of structured and unstructured data accumulated every day–a trend that shows no signs of abating. What’s more, misaligned incentives can cause many problems and exacerbate others, something that I have seen repeatedly in my years as a consultant.

To be sure,  semantic technologies will help us get our arms around enormous data sets, a subject that I have written about on this site before. This is already happening to varying extents. But technology alone won’t get us into the end zone. No, we’ll need highly skilled and adaptable people–and lots of them.

One of my favorite posts on the matter, Rise of the Data Scientist, was written more than two years ago. Data scientists:

…have a combination of skills that not just makes independent work easier and quicker; it makes collaboration more exciting and opens up possibilities in what can be done. Oftentimes, visualization projects are disjointed processes and involve a lot of waiting. Maybe a statistician is waiting for data from a computer scientist; or a graphic designer is waiting for results from an analyst.

Beyond knowledge of different statistical methods and programming languages, the data scientists will need soft skills. Programming a computer is hardly tantamount to dealing with people issues, especially when those darn misaligned incentives rear their ugly heads in.

Simon Says

More than ever today, competitive advantages are fleeting. Never before have companies been able to adapt as quickly, as evinced by examples such as Google, Amazon, and Apple. At the same time, though, nothing has really changed in the last 20 years. The companies that will be able to sustain high levels of success attract, retain, and develop the best people.

Feedback

What say you?