The ‘Big Data’ Buzz – Revolution or Evolution?

‘Big data’ is THE hot new business buzzword of 2012. In this article I want to explore whether big data requires a complete revolution in the way we use and analyze business information or whether the right approach is more of an evolution from the more traditional approach of using performance data to inform management decision-making. But let’s start by unpicking the big data buzz…

Many predict that the future success of companies – big or small – will depend on their ability to capture, analyze and gain insights from big data. The idea behind the buzz is basically that companies use lager and more complex sets of data to inform their decision-making. This, of course, is not new. Using large volumes of data for decision support has been around for a long time. However, what makes ‘big data’ big is the fact that traditional data warehouse set-ups and business intelligence applications can’t handle it because it is so vast in volume, so fast-moving and varied in formats.

Big data is not really clearly defined yet but it generally refers to the more messy types of data that we can’t easily put into rows and columns and that are too big to store and analyze in our traditional data systems. Examples of big data include the vast and ever changing amounts of data generated in social networks where we have conversations with each other, video or photo data, internet search and browser logs, as well as the ever-growing amount of data generated by the sensors and chips in our smart phones and tablets. Using big data would, for example, allow companies to better understand customer or employee behaviors by using their own transactional data and weblogs and combining this with video analysis, social media postings and search engine data to get much richer insights.

Big data is a new business buzz word because of the recent advances in technology, which now enable us to analyze and use larger and less structured volumes of data. Examples of the technological innovations include cloud computing (e.g. providing data storage and access over a network or the Internet), Hadoop (an open source software framework that supports large volume data processing by distributing raw data across different systems) and in-memory data analytics (where data is no-longer stored and queried in disks but instead is processed much faster at runtime).

So in essence, big data refers to leveraging data that was previously seen as unmanageable, which is now possible because of new technologies and approaches to data processing.

While some companies such as Facebook, Google, Amazon and Wal-Mart are already able to leverage big data to inform their decision making and gain competitive advantages (which I describe in much more detail in my book ‘The Intelligent Company’), most businesses are miles away from deriving competitive advantages from big data. In fact, most companies are still struggling with collecting, analyzing and interpreting traditional data, let alone big data.

So, do we have a big data revolution were we through over the old and bring in the new? Or is it more about evolution where we build on the old but improve and enhance it with the new? For me the rise and availability of big data analytics should be seen as an evolution where we build on good traditional data analysis and enhance it with big data where possible and appropriate.

The big data possibilities are so big that we can so easily fall into the same (or even bigger) traps and bad habits that we have fallen into with traditional data analysis: where we collect and analyze everything that walks and moves and nothing that matters.

With big data analytics it is even more important than ever that we follow the basic principles of good analytics, which are:

Be sure your big data strategy clearly aligned to your business strategy
Be very clear about the questions you want to have an answer to
Be very clear about how you will use the data and insights to inform your decision making
And only use big data analytics if there is a clear business case for it.

For me the evolution towards big data analysis starts with the basics of better data-driven decision making and more often than not these principles have to be established using more traditional ‘small data’ and performance metrics. Once the basics are in place companies can move to big data without the danger of falling over at the first hurdle.

More soon, in the meantime, feel free to follow me on Twitter #bernardmarr or lean more about being an intelligent company.

Bernard