The Case for Data Hoarding

December 17, 2012
45 Views

Do we really need all of this data?

It’s a question that I’ve heard more than a few times in my consulting career, especially as organizations have moved from legacy systems to more contemporary equivalents. Most of these improvements mean that organizations can consolidate data sources and, at least in theory, store all of their data in one place.

Do we really need all of this data?

It’s a question that I’ve heard more than a few times in my consulting career, especially as organizations have moved from legacy systems to more contemporary equivalents. Most of these improvements mean that organizations can consolidate data sources and, at least in theory, store all of their data in one place.

Now, generally speaking, hoarding is not a healthy thing–and I don’t need reality shows to tell me as much. With respect to data management, though, is it really detrimental to an organization?

Amber Simonsen seems to think so. Simonsen is the PMP Executive Project Manager of Pierce Transit and talks about the perils of data hoards. ”What begins as an innocent desire to keep relevant information close at hand can turn into an unhealthy obsession that plagues IT departments and Records Managers in organizations everywhere,” Simonsen warns.

It’s a fair point, but think for a minute about data storage costs. To say that they’ve dropped in cost over the last three decades is the acme of understatement. Consider the following chart:

For more on the methodology used to derive these numbers, click here.

Considerations

So, the cost of data hoarding has dropped exponentially, even as the amount of data available has also risen exponentially. If you do the math, you’ll discover that organizations can store much more information these days than even five years ago for significantly lower costs. As a result, it’s hard to buy the “it’s too expensive to store this data” argument.

Of course, just because an organization can store a great deal of data doesn’t mean that it should. Data storage is a continuum, not a binary. What’s more, CIOs can actively decide not to store certain types of information for all sorts of reasons. Perhaps the most important consideration for an organization is what it uses to access and analyze new forms of data. If you’re trying to cram Big Data into Small Data solutions, then data gathering and storage (never mind) hoarding is going to pose a significant problem. As I write in my new book, you can’t write SQL statements against petabytes of unstructured data and expect to see meaningful results. New tools are needed to make sense of Big Data.

Simon Says

Data hoarding only makes sense if two conditions are met:

  1. Your organization has deployed the right tools–i.e., Hadoop, NoSQL and columnar databases, etc.
  2. Your organization actually does something with that data.

If that’s the case, hoard away.

Feedback

What say you?

You may be interested

The Race for 5G Is the Race for Data Dominance
Big Data
80 shares711 views
Big Data
80 shares711 views

The Race for 5G Is the Race for Data Dominance

Daniel Matthews - June 22, 2017

Have you noticed how often the phrase “by the year 2020” comes up? In the tech sphere, many are heralding…

The Direst Security Breaches of 2017 and How Data Centers Are Responding
Best Practices
108 shares1,391 views
Best Practices
108 shares1,391 views

The Direst Security Breaches of 2017 and How Data Centers Are Responding

Diana Hope - June 20, 2017

Cybersecurity is becoming a tremendous concern. By 2021, security breach cost will exceed $6 trillion a year. A number of…

Why Smart Data is the Key to Future Lending
Best Practices
55 shares1,162 views
Best Practices
55 shares1,162 views

Why Smart Data is the Key to Future Lending

Patrick Köck - June 20, 2017

Last month, big data and investment trading collided on the front page of The Wall Street Journal. The article titled…