The Humbling Effect of Big Data

Many of us walk around with the knowledge that we generally understand how the world works. When people ask us for our professional opinions, we’re often more than happy to oblige. Some of us are even flattered when someone wants to know what we should do when faced with uncertainty.

Many of us walk around with the knowledge that we generally understand how the world works. When people ask us for our professional opinions, we’re often more than happy to oblige. Some of us are even flattered when someone wants to know what we should do when faced with uncertainty.

But do we know as much as we think we do? I have my doubts, especially after finishing Everything Is Obvious: How Common Sense Fails Us by Duncan J. Watts. It’s a thought-provoking book because it challenges readers to ask themselves, “What do we really know?” The answer, unfortunately, is not as much as we think.

Simple vs. Complex Systems

As Watts points out, part of the problem stems from the distinction between simple and complex systems. Let’s take a simple situation: the game of blackjack. I’ve been known to play a few hands, and only a fool would hit on 18 with the dealer showing a six. It’s just bad strategy. In such a simple environment, the best decision is clear.

The problem with simple systems is that they aren’t terribly representative. In complex systems like, say, the economy, myriad forces are at play, the vast majority of which are not under the control of any one person, group, department, or organization. Even the US government could not completely solve the financial crisis with a $900 billion stimulus. Bottom line: there’s only so much that any of us can do in a complex system.

And, as I turned the final pages on what is easily the best book I’ve read this year, I started to think about that big question in the context of Big Data. If we truly embrace Big Data, then we will have to question many long-held assumptions about how many things work: our jobs, our departments, our industries, and our environments. Looking at data with open eyes means that we may not like what we see, nor what that data will tell us. And that makes many of us uncomfortable. How many of us want to constantly question what we think we know?

Simon Says

To me, Big Data is not all about technology. Far from it. I’d argue that there’s a human element in all new technologies, and Big Data is no exception to this rule. There’s an organizational readiness component to it, as well as a personal one. Will people and organizations unaccustomed to consulting the data suddenly change their behavior? Will they be open-minded? Or will they act as if they know how things have worked, work now, and will work in the future.

It’s a big question. Big Data may prove that you don’t know as much as we think you do. What will you do then?

Feedback

What say you?