Big Data moves up the stack

The Big Data field itself is rapidly evolving, maturing and broadening in focus.  It is still going through the process of finding itself, working out what it is supposed to be.  While 12 months ago Big Data was, to me at least, a categorization for the platforms that provided data scalability I think that is less so today.  Big Data is becoming more about the layers built on top of those platforms and the value added to the data in those layers.  This is not an unexpected move, it follows path in the direction of the data-as-a-service vision that I and others have shared for some time.

I see this shift being reflected in the companies that are finding success.  While true killer innovation will almost always find funding, killer innovation today often has to be more than just n+1 scalability.  Some companies I know that have built “faster transaction processing” or “more scalable analytics” have found getting a foothold difficult in a crowded market.  The “more scalable” mantra on its own is starting to not be enough to gain and keep attention.  So many platforms in both transaction processing and analytics (both SQL & NoSQL) are delivering high scalability today.  Many of these are open source, and on the closed source side of the fence it appears consolidation needs to happen for sustainability.  Some has happened already and I expect more will follow. 

Image by Délirante bestiole [Lumpen river] via Flickr

I think moving up the stack provides some clear air.  A unique point of difference based around the value added to the underlying data seems to me to offer a more clearly defined proposition.  A unique Big Data platform may be built in the process, but how that platform is applied to enrich information can be more interesting than the platform itself.

Don’t get me wrong.  Killer innovation in Big Data layers form the hardware to the user are important (flash, hadoop, MPP etc) and should continue to exist in their own right.  But a difficulty is launching a Big Data platform in a busy space means the platform may only get a small following.  And platforms with small followings, I think, are difficult to sustain.