Information overload and innovation

I often hear people talking about the amount of data that is being created as being unprecedented.  It isn’t.  What is unprecedented is its retention.  We are all conditioned to the “growth of data”.  This is really lazy language and should be changed to the “growth of the retention of data”.  A business process which creates data has probably been always creating it, but until recently it was probably just transient.

I often hear people talking about the amount of data that is being created as being unprecedented.  It isn’t.  What is unprecedented is its retention.  We are all conditioned to the “growth of data”.  This is really lazy language and should be changed to the “growth of the retention of data”.  A business process which creates data has probably been always creating it, but until recently it was probably just transient.

Modern business has evolved from the industrial revolution.  The problem we face today in navigating the information revolution is that the industrial revolution taught us to use the principles of processes.  Two centuries of business has slavishly adhered to the idea that commercial and government enterprises are nothing more than the aggregate output of thousands of individual business processes.  Because no-one alive today has experienced any other form of business interaction we can be forgiven for thinking that there is no other alternative.

In a previous post, I talked about the use of the “Small Worlds” measure to test innovation (see the post and the introduction to the use of Small Worlds).  Before the information revolution began, an innovation could only be tested in terms of the processes that it affected.  Today the best way to determine if what is being proposed is actually new is to measure the use of data in new business models:  But what does this mean in terms of new data?

I used the example of Amazon recommending books for your future purchase. The data that they use isn’t new, you have always had an identity and you always made individual book purchases, it is just that it wasn’t previously kept beyond the time of the transaction.

While Amazon isn’t really creating new data, some business innovations are actually creating something that didn’t exist before.  Consider the creation of a loyalty scheme by an airline, although the initial interaction is the same a new interaction is generated, which is the redemption, which provides new information.  In this case the new process is an innovation measured by the quantity of information (increased) but also by the tightness of the connections within the data (as measured through the Small Worlds measure).

I argue that although the creation of new data in absolute terms (as opposed to the retention of existing data) means the innovation is genuinely new, it does not become disruptive to existing business unless it actually enhances the connections to current data.  Creating new data on its own doesn’t add much value to an existing business, but creating more links definitely does.

Business and government innovation is best measured by the new connections it adds to society and the organisations that support it rather than by the quantity of transient data that becomes persistent or even the amount of truly new data.  Adding something new adds the greatest value to the people that it serves when it increases the number of connections.

Microblogging, e-health and smart ticketing are all examples of something new and innovative.  When you examine each of them, their real value is not in the creation of data but rather in the connections they generate.  Twitter really penetrates our online activities through the hashtags.  E-health provides links between existing service providers.  Smart ticketing allows transport operators to connect their usage information with their resource planning.