The Scourge of Data Silos

“Those who cannot remember the past are condemned to repeat it.” [1]

“Those who cannot remember the past are condemned to repeat it.” [1]

Over the years there have been many technology waves related to the design, development and deployment of Business Intelligence (BI).  As BI technologies evolved, they have been able to significantly expand their functionality by leveraging the incredible capacity growth of CPUs, storage, disk I/O, memory and network bandwidth. New technologies have emerged as enterprises’ data needs keep expanding in variety, volume and velocity.

Technology waves are occurring more frequently than ever. Current technology waves include Big Data, data virtualization, columnar databases, BI appliances, in-memory analytics, predictive analytics, and self-service BI.

Common Promises

Each wave brings with it the promise of faster, easier to use and cheaper BI solutions. Each wave promises to be the breakthrough that makes the “old ways” archaic, and introduces a new dawn of pervasive BI responsive to business needs. No more spreadsheets or reports needed!

IT and product vendors are ever hopeful that the latest technology wave will be the magic elixir for BI, however, people seem to miss that it is not technology that is the gating factor to pervasive BI. What has held back BI has been the reluctance to address the core issues of establishing enterprise data management, information architecture and data governance. Those core issues are hard and the perpetual hope is that one of these technology waves will be the Holy Grail of BI and allow enterprises to skip the hard work of transforming and managing information. We have discussed these issues many times (and will again), but what I want to discuss is the inevitable result in the blind faith in the latest technology wave.

Common Results

Each wave typically advances BI in some way, but does not really create the breakaway promised; truly pervasive BI remains the Holy Grail. What is almost guaranteed with each new wave, however, is yet another data silo added to the many that already typically litter an enterprise landscape. As a result, many enterprises have one or more of the following: multiple data warehouses, operational data stores (ODS), independent (silo’d) data marts, spreadmarts, data shadow systems and scattered databases used for reporting. Further evidence of the tech wave spawning silos is that enterprises typically have multiple BI product implementations with six or more different BI products.

The people involved in each new wave typically assume that their predecessors (who implemented a previous tech wave) did not know what they were doing and/or just did not use the right technology. They think they know best, and the previous generation (maybe just a few years from being the latest wave) has nothing of value to teach them. Hence, the newest generation does it all from scratch, encountering the same pitfalls – pitfalls that often have nothing to do with technology and everything to do with data management, information architecture and data governance.

Sometimes, they create a new solution separate from the existing BI environment. After all, why get bogged done in the old wave, now a legacy application, when you can start from scratch. This means the next tech wave builds a new tech silo. It may produce business value in the short run, but in the long run the business group and IT have another silo to reconcile.

Enterprises need to look at their Big Data, data virtualization, columnar databases, BI appliances, in-memory analytics, predictive analytics, and self-service BI efforts and determine if they are going to be the next data silos. Unfortunately, the answer is likely to be yes.

The bottom-line is that business people need data and analytics from across the enterprise and not just from a new shiny BI silo. But for this to happen, the BI team has to stop the madness by not creating yet another silo and legacy application. Can you say “no” to the data silo?