What is the biggest challenge for Big Data? (5 years on)

Five years doesn’t half fly when you’re having fun!  In this post from 2011 I highlighted some of the challenges facing the “big data revolution” centring on a lack of people with the right skills to deliver value on the proposition.  Fast forward to 2016 and this not only remains true, but is likely the key issue holding back the adopting of advanced analytics in many organisations.

 While there has been an influx of “Data Scientist” titles across the industry, generally organisations are still adopting a technology driven approach driven by IT.  The conversations are still very focused on the how rather than the why, it is still all very v1.0.  There is still a lack of the knowledge required to turn potential into value, value that directly affects an organisations bottom line.

This will start to sort itself out as the field matures and those who understand the business side of the coin become fluent with big data concepts, to the point they can direct the engineering gurus.  IBM with Watson is looking to take this a step further by bypassing the data techies and letting analysts explore data without as much consideration for the engineering/plumbing involved.  This is a similar direction that services such as AWS and Azure Machine Learning are heading, in the cloud.

In 2016 the biggest challenge for Big Data is turning down the focus on the technical how, and turning up the focus on the business driven why.  Engaging and educating those who understand a given business in the capabilities of data science, motivating them to lead these initiatives in their organisations.