I have been working in analytics for over twenty years, and have witnessed first hand how these technologies have made the world a better place. I’ve seen thousands of examples, from every type of corporate efficiency imaginable, to improving customer satisfaction at theme parks and making better use of limited blood supplies.
Yet we’ve only seen the tip of the iceberg when it comes to “big data analytics.” There are many different definitions of this term, but my favorite is “using data that was previously ignored because of technology limitations” — and those limitations are falling away fast, leading to cascades of new opportunities.
We can now gather, correlate, and analyze information in ways that were unthinkable in the past. The book “The Human Face of Big Data” does a particularly good job of rounding up some of the most interesting stories of how these technologies will touch our lives.
However, with great power comes great responsibility. Analytics is a very powerful weapon, and weapons can be abused.
The past clearly shows that without proper controls, there can be irresistible temptations for companies and governments to combine data in ways that threaten personal liberties. Misuse of every previous data gathering technology has eventually come to light, sometimes only decades after the facts, leading to new laws re-establishing privacy limits.
Modern technology makes the potential threat much greater than in the past. Combining “metadata” from online activities, mobile devices, payment systems, surveillance cameras, medical histories, and social networks can reveal every nuance of our online and offline lives.
While nobody is arguing that the Thought Police exists today, for the first time since I read George Orwell’s “1984” thirty years ago, we have their technology capabilities:
“There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to.”
Analytics is, at best, a wonderful opportunity to shine light into the dark, to reveal what was previously concealed, and make it better. People and governments must be in the forefront of establishing clear, transparent guidelines that make the right tradeoffs between the public good and citizen’s rights.
We should not wait for abuses to come to light before acting. One common refrain that defends data gathering is “if you’re not doing anything wrong, what are you worried about?” – if this were true, it should surely also apply to the organizations currently storing and using our data in “secret” ways today.
I believe the vendors of analytics software should play a part in this debate, help encourage data safety and transparency, and provide the technology features that makes it easy for organizations to support these initiatives.