The Future of Data and Analytics – Bigger and Faster

When you’re talking about bytes, mega is “so yesterday” and peta is quickly going the way of “kilo.” More is better in lots of ways but in forecasting the impact on data analytics and business intelligence, you’ll find the units of measure just keep getting bigger. Instead of terabytes in your data warehouse, get ready for exabytes (1 billion gigabytes) or yottabytes (one quadrillion gigabytes). No wonder Information Week devoted a recent cover story to “Big and Fast” – the need for getting insights from huge data sets with ever-increasing speed and accuracy. 

To us, the key idea in “Big and Fast” is the concurrence of both adjectives – by working in conjunction are going to dramatically remake data analytics as tools improve and uses proliferate. Databases will only get larger, more complex and more interconnected. Doug Henschen reports that nearly half of all organizations surveyed by the Data Warehousing Institute – 46 percent of them — plan to change their data warehousing platform by 2012.

The online resource center offers advice from eight companies – Cabela’s to Catalina Marketing to Barnes & Noble – that have hundreds of stores and millions of online purchase records. In-database modeling and other features that Spotfire users already know about, permit even large data to be used more efficiently. We can’t wait to copy data out of source warehouses, transfer it to a new application, perform a task and then return an answer. If demands for Web bandwidth are any clue and Google can plow through searches and deliver a set of results in under a second, then the expectation of speed AND power is only going to increase the pressure.

Henschen charts the future including the replacement of disk drives with solid-state drives (SSD) a storage technology that can produce speeds of 150x the read/write/retrieve times of disks. Yet costs that can be 10x the expense of disk technology, mean it will take years for data centers to make the switch. Laptops with SSD instead of disks are an early indicator. Some manufacturers tout the lightweight, high-speed, reliability and durability that are coming our way.

Ready to face the future? You’ll want to practice pronouncing zettabyte and yottabyte — both measures of data will eventually replace exabyte. One study by Cisco contends the global monthly Internet traffic as of March 2010 is estimated to be 21 exabytes. However, zettabtye and yottabyte data amounts have yet to be approached. When we reach those measurements at least you’ll know the language … And hopefully you’ll have data analytics software that can keep pace.

David Wallace
Spotfire Blogging Team