The Big Data field itself is rapidly evolving, maturing and broadening in focus. It is still going through the process of finding itself, working out what it is supposed to be. While 12 months ago Big Data was, to me at least, a categorization for the platforms that provided data scalability I think that is less so today. Big Data is becoming more about the layers built on top of those platforms and the value added to the data in those layers. This is not an unexpected move, it follows path in the direction of the data-as-a-service vision that I and others have shared for some time.
I see this shift being reflected in the companies that are finding success. While true killer innovation will almost always find funding, killer innovation today often has to be more than just n+1 scalability. Some companies I know that have built “faster transaction processing” or “more scalable analytics” have found getting a foothold difficult in a crowded market. The “more scalable” mantra on its own is starting to not be enough to gain and keep attention. So many platforms in both transaction processing and analytics (both SQL & NoSQL) are delivering high scalability today. Many of these are open source, and on the closed source side of the fence it appears consolidation needs to happen for sustainability. Some has happened already and I expect more will follow.
Image by Délirante bestiole [Lumpen river] via Flickr
Don’t get me wrong. Killer innovation in Big Data layers form the hardware to the user are important (flash, hadoop, MPP etc) and should continue to exist in their own right. But a difficulty is launching a Big Data platform in a busy space means the platform may only get a small following. And platforms with small followings, I think, are difficult to sustain.
