2012 begins to wind down. Yes, I know it’s still only mid-November, but I find it hard to avoid thinking of year-end when the retail industry has been pushing Christmas for weeks already. I’ve been preparing for my keynote at Big Data Deutschland in Frankfurt (20-21 Nov) next week, so it seemed appropriate to share some thinking on where big data is at now. Also, I’ve been deeply involved in analyzing the results of the EMA / 9sight big data survey which has just been published. My bottom line? Big data is dead!
Of course, I don’t mean that literally. What I’m really trying to do is to get the attention of the marketing folks who have been using and abusing the term, particularly during 2012. Two very clear results emerge from the big data survey when it comes to real customer projects carrying the moniker big data.
First, the industry has been besotted by size. Carefully avoiding now all vaguely salacious phrases, the fact is that size is so relative that calling data big or small is more about bragging or shaming than any measure of real use. Our survey showed that 60% of respondents were managing less than 100TB of data in total in their organizations, while only 5% stretched beyond a petabyte. Not all of this data was part of their big data projects; on average, only some 30% was included there. This strongly suggests that so called big data technology is being widely used for something other than processing excessively large data volumes.
Second, it’s not all about exotic types of data either. Yes, some 45% of the data sources fall under the category of human-sourced information, which includes social media sources. But, just over 30% is process-mediated data — transactional data gathered and created in traditional operational and informational applications. For a more detailed explanation of these data domains, as I call them, please see my recent White Paper “The Big Data Zoo – Taming the Beasts, The need for an integrated platform for enterprise information”. So, big data projects are addressing a substantial proportion of the data we’ve known and loved for many years.
You can hear more of the survey results on the EMA / 9sight webinar on Thursday, 13 December, 11 a.m. PST / 2 p.m. EST.
What is actually becoming important as we look towards 2013 is what businesses are really doing with data at the moment that is different from what they’ve traditionally done. I believe there are two distinct trends. One is, of course, business analytics. This is simply an evolution of traditional BI, with more of an emphasis on exploration (or mining) and less on reporting and dashboards. The second is more interesting and, potentially, game changing. This involves the re-integration of operational action taking and informational decision making in customer-facing applications that automatically modify their behavior in real-time in response to rapidly changing market or personal circumstances.
All this says to me that big data as a technological category is becoming an increasingly meaningless name. Big data is essentially all data. Is there any chance that the marketing folks can hear me?
