The Data Lake Debate: The Introduction

In his memoir, Life, Keith Richards dispels the myth that the Rolling Stones had trouble coming up with new songs. If anything according to the legendary guitarist, the band had too many song ideas. “Bent Green Needles,” “Might as Well Get Juiced,” and “Pommes de Terre” were a few that never got much radio rotation, arguably for good reason.

(Thank God “Gimme Shelter” made the cut! #bestsongever)

But how did they know? Sure there were arguments among the band members. Imagine Keith sprawled on the shag carpet at their producer’s Laurel Canyon bungalow, plastic tumbler full of “nuclear waste” (vodka mixed with orange and cranberry juices with a splash of Fanta—Keith’s drink of choice), quarreling with Mick about whether “Hot Stuff” was in or out.

But wait, this is about data, right? Exactly. Because how do we know—with the vast and teeming volumes of information being both consumed in and generated by our companies—how do we know what’s going to work and what isn’t? It was hard enough when data was structured and hypotheses were strong. But in the new age of big data, and the growing popularity of so-called data lakes, separating the valuable data from the inconsequential data requires more than just a stiff drink and an argument.

The question here is whether filling up your data lake will help or hurt the cause. On the one hand, a data lake full of raw, multi-structured, and heterogeneous data from across systems and business processes, could be the proverbial “single version of truth” that up until now had just been the unconsummated hope of many an executive. On the other hand, we might simply be creating another island of information (maybe more of an archipelago) that will house yet more versions of data that is already extant on other core systems. Only this time, business users need to know what to do with it when they have it.

Go ahead and watch this 8-week debate unfold. You might decide in the course of reading what two data-savvy pioneers, Tamara Dull and Anne Buff, have to say that your company needs some place to store all that data you’re dredging up. Or you might decide that you’ll only keep the data that’s meaningful and useful—when compared with the lake, a mere puddle.

Whichever path you take, the time has come for a debate on the data lake, for the simple reason that we’re reminding everyone that formalizing policies around key data yields progress. Delivering a data lake might not satisfy every requirement but…wait for it…if you try sometimes, you get what you need.