Recently I bought a Maxtor OneTouch™ 1TB backup drive for my son’s PC. We had been using a 320 Gb hard drive for backups, but to keep a couple of full image backups with daily incrementals we were maxing out that drive. It is not surprising with the music, graphics files, games, social networking and maybe even a few homework files he would be the first in the family to need a 1TB backup drive.
When I was running later in the day I got to thinking about the fact that I just purchased a 1TB hard drive for a home PC! Not to age myself too much, but when I built my first data warehouse and data mart 21 years ago we had a disk farm of 150 Gb for the entire project. That disk farm consisted of 150 1-Gb drives which, at that time, was the largest that my high tech employer built. We thought the storage that we had available was abundant for the needs of our company which had $10 billion in sales.
Then I thought about how not too many years ago 1TB data warehouses were at the high end of corporate efforts and many of us were impressed with the sizable amount of corporate data that was being made available to businesses.
The good news is that data warehousing and business intelligence has been very successful at many companies, but the bad news is that business needs keep expanding beyond what we have built and beyond what we thought previously was a massive amount of data.
No sooner did we get data from ERP (enterprise resource planning), CRM (customer relationship management) and other enterprise applications that we had to reach out beyond the firewall to suppliers, partners and customers. We have started to tap the data shadow systems (or spreadmarts) that exist in huge pockets in most companies. And we have not really begun to target the untapped, but high value, unstructured data that business people use daily to make decisions.
A 1TB backup drive for a home PC! This indicates about how far we have come – this would have seemed to be enough disk storage in the past to store an entire enterprise-wide data warehouse (DW) but now it’s just enough to backup a home PC – and how much more business data we have yet to incorporate into our data backbone for analytics, performance management and operational reporting.
