The Problem with the Relational Database

The relational database has been the core mechanism for structured data storage and retrieval for the past 30 years.  My career so far has focused around the relational database, whether it be from a development, administrator or investment perspective.  In all this time the RDB has been the best generic option available for developers building data centric applications.  The generic nature of the RDB has made it suitable for wide mix of application requirements, be they heavily transaction processing orientated or heavily data analytics related.

However over the few years we have been witnessing a slow shift aware from the “RDBMS” for everything trend that we saw over the preceding decade.  And this is occurring because the demands we are placing on data in terms of scale and volume are growing to a point where the most generic platform is underperforming and instead more specialist database technologies are starting to be selected based on their closer fit with the requirement.

This trend has started in and is therefore more visible in the data analytics space.  The specialist solutions have be slowly cropping up over the last 5 years and now today it wouldn’t be that unusual for …

The relational database has been the core mechanism for structured data storage and retrieval for the past 30 years.  My career so far has focused around the relational database, whether it be from a development, administrator or investment perspective.  In all this time the RDB has been the best generic option available for developers building data centric applications.  The generic nature of the RDB has made it suitable for wide mix of application requirements, be they heavily transaction processing orientated or heavily data analytics related.

However over the few years we have been witnessing a slow shift aware from the “RDBMS” for everything trend that we saw over the preceding decade.  And this is occurring because the demands we are placing on data in terms of scale and volume are growing to a point where the most generic platform is underperforming and instead more specialist database technologies are starting to be selected based on their closer fit with the requirement.

This trend has started in and is therefore more visible in the data analytics space.  The specialist solutions have be slowly cropping up over the last 5 years and now today it wouldn’t be that unusual for an organization to choose a specialist data analytics database platform (such as those offered from Netezza, Greenplum, Vertica, Aster Data or Kickfire) over a generic database platform offered by IBM, Microsoft, Oracle or Sun for housing data for high end analytics.

My argument is that while I see the traditional generic RDBMS remaining the platform of choice for most generic application requirements in the foreseeable future two breakaway alternative paths are also emerging.  The first is that I mentioned above, a reduction in the generic aspects of the RDBMS with a specific focus on high end data analytics functionality.  The second, which I see starting to emerge right now, is the opposite of this.  A reduction in the generic nature of the RDBMS with a focus on the specific requirements of high end transaction processing.

Starting tomorrow I will be presenting a series of posts that discuss real world issues facing the RDBMS when used in transaction processing environments that are being encountered today to highlight why this alternative path in transaction processing is appearing then following this I will present a series of posts on the technology that is emerging in an attempt to address these weaknesses.

Link to original postInnovations in information management