By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    football analytics
    The Role of Data Analytics in Football Performance
    9 Min Read
    data Analytics instagram stories
    Data Analytics Helps Marketers Make the Most of Instagram Stories
    15 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    What to Know Before Recruiting an Analyst to Handle Company Data
    6 Min Read
    AI analytics
    AI-Based Analytics Are Changing the Future of Credit Cards
    6 Min Read
    data overload showing data analytics
    How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: How Good Management Can Produce Bad Data
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Inside Companies > How Good Management Can Produce Bad Data
Inside Companies

How Good Management Can Produce Bad Data

RamaRamakrishnan
Last updated: 2010/11/15 at 3:23 PM
RamaRamakrishnan
6 Min Read
SHARE

As a long-time analytics practitioner, I am well aware of the dangers of using data without fully understanding where it comes from and how it is generated.

There are numerous ways in which the data one gets isn’t quite what it seems: the same data item may be named differently in different systems, different items may have the same name, the data item may be defined in ways subtly different from what commonsense may indicate etc.

As a long-time analytics practitioner, I am well aware of the dangers of using data without fully understanding where it comes from and how it is generated.

More Read

database compliance guide

Four Strategies For Effective Database Compliance

5 Big Data Storage Solutions
How To Keep Your Data Security Knowledge Up To Date?
Crucial Advantages of Investing in Big Data Management Solutions
Big Data Strategies Hinge on Using Drop Tables in SQL Servers

There are numerous ways in which the data one gets isn’t quite what it seems: the same data item may be named differently in different systems, different items may have the same name, the data item may be defined in ways subtly different from what commonsense may indicate etc.

All these (and more) are well-known issues that surround data and explain why seasoned analytics experts claim that the lion’s share of an analytics project is likely to be data-cleansing, transformation etc. rather than modeling.

But recently I came across an unusual source of bad data: good management.

We have been working with a retailer on ways to assign their store point-of-sale transactions to households so that we can analyze a family’s purchase patterns across multiple shopping trips.

A common problem in this sort of exercise is the need to group individual shoppers into households. Since different members of a household may have different names, credit cards etc,. the customer is often asked for a phone number at the checkout by the cashier who’s ringing up the sale.

Using third-party databases of landline numbers and mobile numbers, we can identify which of the supplied phone numbers is a landline number. Armed with this, we can collect all the transactions with the same landline number and infer that all these purchases were made by the same household.

Can you spot the weak link in this straightforward scheme?

The cashier has to remember to ask the customer for their phone number. It is extra work for the cashier and when there’s a long line of impatient customers in front of you, it is easy to forget.

So what do we do? Incentives to the rescue!

Management sensibly (after all, they were heeding the legendary Peter Drucker’s advice: “What gets measured gets managed”) decided to give store associates a cash bonus based on how many phone numbers they captured.

As expected, the phone number capture rate went up after the incentives were put in place and the retailer was able to assign many more transactions to households than before.

But we noticed some oddities:

  • Some households visited a single store twenty or thirty times a day!
  • Some households had several hundred store transactions annually!

We studied these odd cases and discovered something interesting: these “crazy shopping” households were really dozens of households rolled into one! The reason these distinct households were grouped together were because they had a common phone number.

And  how did they end up with a common phone number?

Because the cashier who rang up their purchases punched in the same phone number for everyone.

Perhaps these customers declined to supply a phone number, perhaps the cashier neglected to ask, who knows ….

Whatever the reason, for a small number of cashiers, it was just too tempting to simply punch in a fake phone number and make their bonus rather than do the right thing.

After this came to light, the retailer was able to mitigate this “phone number fraud” by first cross-checking every entered phone number against a list of store phone numbers and cashier phone numbers etc. This  helped and was a good first step but it is not enough. We are continuing to refine the fraud mitigation algorithm using data mining techniques.

What did I learn from this experience?

I have resolved that whenever I am working with data that was created by people (rather than produced by machines), I will try to understand if the data may be distorted by incentives affecting the behavior of the person(s) creating the data.

And the next time a cashier is ringing up your purchases in a store, see if he/she is entering what looks like a 10-digit number without even asking you :-)

TAGGED: data management
RamaRamakrishnan November 15, 2010
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Shutterstock Licensed Photo - 1051059293 | Rawpixel.com
QR Codes Leverage the Benefits of Big Data in Education
Big Data
football analytics
The Role of Data Analytics in Football Performance
Analytics Big Data Exclusive
smart home data
7 Mind-Blowing Ways Smart Homes Use Data to Save Your Money
Big Data
ai low code frameworks
AI Can Help Accelerate Development with Low-Code Frameworks
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

database compliance guide
Data Management

Four Strategies For Effective Database Compliance

8 Min Read
Data Management

5 Big Data Storage Solutions

6 Min Read
keep data security up to date
Security

How To Keep Your Data Security Knowledge Up To Date?

5 Min Read
benefits of big data management solutions
Big Data

Crucial Advantages of Investing in Big Data Management Solutions

8 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?