Where Did the ‘Data Explosion’ Come From?

ReadWriteWeb, 2010:

One of the key aspects of the emerging Internet of Things – where real-world objects are connected to the Internet – is the massive amount of new data on the Web that will result. As more and more “things” in the world are connected to the Internet, it follows that more data will be uploaded to and downloaded from the cloud. And this is in addition to the burgeoning amount of user-generated content – which has increased 15-fold over the past few years, according to a presentation that Google VP Marissa Mayer made last August at Xerox PARC. Mayer said during her presentation that this “data explosion is bigger than Moore’s law.”

(For those of you that are wondering, “Moore’s law” refers to the theory that data doubles every 18 months.)

After recently stumbling across several articles mentioning the “data explosion” (notably one at information-management.com) it got me thinking about the masses of data being created every second in the world today. In the info-mgmt article they had a couple of interesting paragraphs about the causes of the data explosion, and how to cope with it.

Key take-aways:

In today’s 24×7 business world with demand for timely and relevant information, devising a forward-looking big data strategy is critical to ensure your organization can effectively leverage data from all available sources and quickly turn it into a competitive advantage
Online and offline behavior are blending into a single entity.
There are many sources of data that are contributing to this remarkable data growth.
Data comes from mobile, Internet and traditional sources… and people are evolving from information consumers to producers, by creating their own data.
Common but often ineffective coping methods: adding more hardware, pushing data transformations elsewhere, such as down into the database, or custom coding when addressing data performance problems that arise as data volumes grow.
Overall objective: quickly adapt and respond to changing business demands.
Integrating enterprise data and new sources of data (e.g. web data) is the way forward for timely, accurate access to information as a basis for making business decisions.
To manage big data for the long-term as data volumes continue to grow, look to create a fast, efficient, simple cost-effective data integration environment.
With a sound strategy in place, big data can actually help provide the key to unlocking an organization’s next big opportunity.

So what are some reasons for the data explosion?

1. A major trend over the last few years has seen many organisations implementing ERP and CRM solutions. This in turn has caused a dramatic increase in the amount of data we are storing about our customers, prospects, partners and suppliers.

2. The current economic state of the world is likely to have resulted in a higher number of business deals, such as mergers and acquisitions. This in turn has resulted in growing data volumes, with the associated issues such as duplication of information. Organizations therefore have to not only manage all of their own data, both old and new, but also this wave of incoming data from other sources.

3. Legislation changes could also have played a major part in the retention of data. The laws have changed around how we use, store and maintain our data. Organizations are often required to hang on to more data, for longer.

4. The improvements in technology and database capabilities have changed the way data is stored and scaled, and the structure and format of the data. New innovations have enabled more and more data to be managed and supported, while improvements in speed, quality and storage are being made all the time.

Coping with the data explosion

Be open-minded to change. A typical response is simply to add more hardware to the system, more storage, more servers and more bandwidth, but this does not generally solve the issue – in fact it can contribute to capturing more data, and thereby increasing the cost of maintenance and storage with more servers, IT staff and bigger facilities to house it all. Find out what works for you and your organization best. For example – choosing your data storage. Relational databases are ideally suited to the task of capturing and storing huge quantities of data and are designed to cope with a massive throughput of transactions. Then you can choose an analysis tool that compliments your already existing business intelligence investment.