We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
    chatgpt image jul 21, 2026, 04 34 30 pm
    4 Core Benefits of Predictive Maintenance after Vibration Analysis
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: How Good Management Can Produce Bad Data
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Inside Companies > How Good Management Can Produce Bad Data
Inside Companies

How Good Management Can Produce Bad Data

RamaRamakrishnan
RamaRamakrishnan
6 Min Read
How Good Management Can Produce Bad Data
Photo by CUsai on Pixabay (https://pixabay.com/photos/the-labour-code-human-ressources-3520805/)
SHARE

As a long-time analytics practitioner, I am well aware of the dangers of using data without fully understanding where it comes from and how it is generated.

There are numerous ways in which the data one gets isn’t quite what it seems: the same data item may be named differently in different systems, different items may have the same name, the data item may be defined in ways subtly different from what commonsense may indicate etc.

All these (and more) are well-known issues that surround data and explain why seasoned analytics experts claim that the lion’s share of an analytics project is likely to be data-cleansing, transformation etc. rather than modeling.

But recently I came across an unusual source of bad data: good management.

More Read

That Wall Dividing IT and Users
That Wall Dividing IT and Users
Is It Possible to Fully Protect Your Data Nowadays?
Business rules, accelerating change and Decision Management
PCI-Compliant Supermarket Chain Bashas’ Breached
Choosing Between Outsourced Vs In-House Data Management Strategies

We have been working with a retailer on ways to assign their store point-of-sale transactions to households so that we can analyze a family’s purchase patterns across multiple shopping trips.

A common problem in this sort of exercise is the need to group individual shoppers into households. Since different members of a household may have different names, credit cards etc,. the customer is often asked for a phone number at the checkout by the cashier who’s ringing up the sale.

Using third-party databases of landline numbers and mobile numbers, we can identify which of the supplied phone numbers is a landline number. Armed with this, we can collect all the transactions with the same landline number and infer that all these purchases were made by the same household.

Can you spot the weak link in this straightforward scheme?

The cashier has to remember to ask the customer for their phone number. It is extra work for the cashier and when there’s a long line of impatient customers in front of you, it is easy to forget.

So what do we do? Incentives to the rescue!

Management sensibly (after all, they were heeding the legendary Peter Drucker’s advice: “What gets measured gets managed”) decided to give store associates a cash bonus based on how many phone numbers they captured.

As expected, the phone number capture rate went up after the incentives were put in place and the retailer was able to assign many more transactions to households than before.

But we noticed some oddities:

  • Some households visited a single store twenty or thirty times a day!
  • Some households had several hundred store transactions annually!

We studied these odd cases and discovered something interesting: these “crazy shopping” households were really dozens of households rolled into one! The reason these distinct households were grouped together were because they had a common phone number.

And  how did they end up with a common phone number?

Because the cashier who rang up their purchases punched in the same phone number for everyone.

Perhaps these customers declined to supply a phone number, perhaps the cashier neglected to ask, who knows ….

Whatever the reason, for a small number of cashiers, it was just too tempting to simply punch in a fake phone number and make their bonus rather than do the right thing.

After this came to light, the retailer was able to mitigate this “phone number fraud” by first cross-checking every entered phone number against a list of store phone numbers and cashier phone numbers etc. This  helped and was a good first step but it is not enough. We are continuing to refine the fraud mitigation algorithm using data mining techniques.

What did I learn from this experience?

I have resolved that whenever I am working with data that was created by people (rather than produced by machines), I will try to understand if the data may be distorted by incentives affecting the behavior of the person(s) creating the data.

And the next time a cashier is ringing up your purchases in a store, see if he/she is entering what looks like a 10-digit number without even asking you :-)

TAGGED:data management
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
Big Data Business Intelligence Exclusive Web Analytics
Flat editorial illustration: A SaaS-editorial scene showing product usage, support patterns, and renewal feedback informing a fle
Using Product Usage Analytics and Support Data to Build a B2B Product Roadmap
Business Intelligence Infographic
From Hiring To Payroll: Managing HR Data Across Business Systems -- AI-generated illustration
From Hiring To Payroll: Managing HR Data Across Business Systems
Big Data Business Intelligence Infographic
Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
Big Data Business Intelligence Exclusive

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Big Data Research
Best PracticesBig Data

Why You Need A Methodology For Your Big Data Research

6 Min Read
Flat editorial illustration: The article explains that file-system choice determines whether Windows, macOS, and Linux can read o
Cloud ComputingExclusive

Understanding File System Compatibility Across Windows, macOS, and Linux

9 Min Read
IoT Security: What Kind of Data Is Compromised by Poorly Protected IoT Devices?
Internet of Things

IoT Security: What Kind of Data Is Compromised by Poorly Protected IoT Devices?

6 Min Read
Examples of Using Kanban Boards with Data Visualization Tools
Data Visualization

Examples of Using Kanban Boards with Data Visualization Tools

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

From Bolts to Bots: How AI Is Fortifying the Automotive Industry
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?