The Use and Abuse of Big Data and Hadoop

Big data analytics is having a huge impact on us all. The ability to collect, store and analyze massive amounts of disparate data—using analytics platforms such as Qubole to uncover hidden connections, correlations and insights—is playing a bigger role than ever in influencing almost every aspect of our lives.

Big data analytics is having a huge impact on us all. The ability to collect, store and analyze massive amounts of disparate data—using analytics platforms such as Qubole to uncover hidden connections, correlations and insights—is playing a bigger role than ever in influencing almost every aspect of our lives. And as the amount of data created every day grows exponentially, it seems that there is no limit to what big data holds for us in the future.

Still, in light of the Snowden and NSA controversies, the idea of more information becoming “too much information” is raising concerns, especially in the area of personal privacy. And although we might chalk up big data abuse to Big Brother, the truth is that a number of organizations that we have come to know and trust—such as the big three listed below—could already be compromising our personal privacy by taking big data too far.

Amazon.com – Starting out primarily as a book retailer, Amazon gained popularity quickly for its massive selection and the ability to deliver what we wanted very quickly. Using algorithms to show us what other customers like us were also buying, Amazon grew rapidly into an ecommerce juggernaut. Today predictive analytics is playing a pivotal role in enabling Amazon to leverage customer data like never before, through a process called “Anticipatory Shipping.” The end game of anticipatory shipping is to accurately select and ship an item to a customer before that customer has even made the decision to buy—all made possible by analyzing the customer’s personal behaviors.

For consumers, the big question with regard to big data and Amazon’s “Anticipatory Shipping” practice lies in just how much power a company should have over their purchasing decisions. After all, shouldn’t consumers be the ones to decide what they buy and when? Isn’t Amazon taking big data too far? In the very near future, if Amazon has its way, we could be receiving goods we never ordered, conveniently delivered to our doorsteps by drones. Hopefully Amazon is also working on algorithms to predict all of the things their customers will be returning so they can promptly send the drones back to retrieve unwanted goods.

Facebook – Developed as a means for people to connect and interact, Facebook has become a big data exploiter’s dream come true. After all, what other site encourages people to willingly post all kinds of personal information about their relationships, interests, habits, hobbies, likes and dislikes, etc.—all in the name of becoming more socially connected? But as we’re now discovering, what Facebook is doing with all that personal information is a little disconcerting.

A few years back, USA Today revealed that Facebook isn’t just analyzing our profile data. It’s actually tracking us across the web via “tracking cookies” that are secretly inserted into our web browsers when we create a Facebook account. This allows Facebook to identify what sites we are visiting, not just when we are logged into Facebook, but apparently when we are logged out as well. And all of this information is up for sale to Facebook’s advertising partners. This invasion of our privacy makes a pretty strong case that Facebook is taking big data too far. Add to this Facebook’s foray into facial recognition—a capability that will allow Facebook to better track what we’re doing and with whom—and the Big Brother implications are downright scary.

Target, Inc. – Companies have been using big data to get a better read on their customers wants and needs for quite some time now. But the questionable big data practices of a local neighborhood Target store literally brings the notion of taking big data too far closer to home. It all surfaced back in 2012 when a teenage girl bought some pregnancy-related items from a Target store in Minneapolis. Due to Target’s aggressive campaign to identify pregnant customers and then recruit them as moms who would buy all of their baby products at Target, the company began sending coupons for diapers and other baby products to the girl’s address. The coupons caught the attention of the girl’s father, who called the Target store to question why they were sending such coupons to his still-in high school-daughter. Long story short, Target knew about the girl’s pregnancy before her dad did, which clearly compromised the girl’s right to privacy. Since the incident, which has taken on urban legend status, Target has taken steps to put pregnancy-related product coupons in front of expectant mothers without letting on that they are basically spying on them.

Like all tools, big data analytics has real benefits for the real world. The human genome project, soon to revolutionize healthcare through the delivery of personalized, gene-specific treatments for life-threatening diseases, is a prime example of how big data, used correctly, is helping mankind. As for enterprises such as those discussed above, it’s easy to see how tempting it is for marketers to use big data in questionable ways, as the benefits to the bottom line are real and substantial. The ongoing challenge for big data analytics—as it is with all technologies—is finding the right balance between the benefits gained by using the technology appropriately and the very real risks incurred by taking things too far.

Image Source:Wikimedia