Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
    pexels pavel danilyuk 8112119
    Data Analytics Is Revolutionizing Medical Credentialing
    8 Min Read
    data and seo
    Maximize SEO Success with Powerful Data Analytics Insights
    8 Min Read
    data analytics for trademark registration
    Optimizing Trademark Registration with Data Analytics
    6 Min Read
    data analytics for finding zip codes
    Unlocking Zip Code Insights with Data Analytics
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: How Good Management Can Produce Bad Data
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Inside Companies > How Good Management Can Produce Bad Data
Inside Companies

How Good Management Can Produce Bad Data

RamaRamakrishnan
RamaRamakrishnan
6 Min Read
SHARE

As a long-time analytics practitioner, I am well aware of the dangers of using data without fully understanding where it comes from and how it is generated.

There are numerous ways in which the data one gets isn’t quite what it seems: the same data item may be named differently in different systems, different items may have the same name, the data item may be defined in ways subtly different from what commonsense may indicate etc.

As a long-time analytics practitioner, I am well aware of the dangers of using data without fully understanding where it comes from and how it is generated.

More Read

byod mobile management
BYOD: Reducing the Risk with Mobile App Management
Data Controls and Customer Loyalty: How Big Companies Keep Clients
Final Jeopardy- Stephen Baker’s quest to know everything
Back To Basics: The Business Requirement For Reporting
Five Ways to Build a Data-Driven Resume and Cover Letter

There are numerous ways in which the data one gets isn’t quite what it seems: the same data item may be named differently in different systems, different items may have the same name, the data item may be defined in ways subtly different from what commonsense may indicate etc.

All these (and more) are well-known issues that surround data and explain why seasoned analytics experts claim that the lion’s share of an analytics project is likely to be data-cleansing, transformation etc. rather than modeling.

But recently I came across an unusual source of bad data: good management.

We have been working with a retailer on ways to assign their store point-of-sale transactions to households so that we can analyze a family’s purchase patterns across multiple shopping trips.

A common problem in this sort of exercise is the need to group individual shoppers into households. Since different members of a household may have different names, credit cards etc,. the customer is often asked for a phone number at the checkout by the cashier who’s ringing up the sale.

Using third-party databases of landline numbers and mobile numbers, we can identify which of the supplied phone numbers is a landline number. Armed with this, we can collect all the transactions with the same landline number and infer that all these purchases were made by the same household.

Can you spot the weak link in this straightforward scheme?

The cashier has to remember to ask the customer for their phone number. It is extra work for the cashier and when there’s a long line of impatient customers in front of you, it is easy to forget.

So what do we do? Incentives to the rescue!

Management sensibly (after all, they were heeding the legendary Peter Drucker’s advice: “What gets measured gets managed”) decided to give store associates a cash bonus based on how many phone numbers they captured.

As expected, the phone number capture rate went up after the incentives were put in place and the retailer was able to assign many more transactions to households than before.

But we noticed some oddities:

  • Some households visited a single store twenty or thirty times a day!
  • Some households had several hundred store transactions annually!

We studied these odd cases and discovered something interesting: these “crazy shopping” households were really dozens of households rolled into one! The reason these distinct households were grouped together were because they had a common phone number.

And  how did they end up with a common phone number?

Because the cashier who rang up their purchases punched in the same phone number for everyone.

Perhaps these customers declined to supply a phone number, perhaps the cashier neglected to ask, who knows ….

Whatever the reason, for a small number of cashiers, it was just too tempting to simply punch in a fake phone number and make their bonus rather than do the right thing.

After this came to light, the retailer was able to mitigate this “phone number fraud” by first cross-checking every entered phone number against a list of store phone numbers and cashier phone numbers etc. This  helped and was a good first step but it is not enough. We are continuing to refine the fraud mitigation algorithm using data mining techniques.

What did I learn from this experience?

I have resolved that whenever I am working with data that was created by people (rather than produced by machines), I will try to understand if the data may be distorted by incentives affecting the behavior of the person(s) creating the data.

And the next time a cashier is ringing up your purchases in a store, see if he/she is entering what looks like a 10-digit number without even asking you :-)

TAGGED:data management
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

crypto marketing
How a Crypto Marketing Agency Can Use AI to Create Powerful Native Advertising Strategies
Blockchain Exclusive Marketing
data driven insights
How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
Analytics Big Data Exclusive
image fx (37)
Boosting SMS Marketing Efficiency with AI Automation
Exclusive
pexels pavel danilyuk 8112119
Data Analytics Is Revolutionizing Medical Credentialing
Analytics Big Data Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

big data for wealth management industry
AnalyticsBig DataData ManagementExclusive

How Could New Big Data Technology Benefit Wealth Management Industry?

6 Min Read

Rating agency data: Getting gamed

3 Min Read
Data Management

Administering Data Fabric to Overcome Data Management Challenges.

8 Min Read

7 Mistakes In Enterprise Data Management You Can’t Afford to Make: Speaking at DAMA WI & DAMA Chicago

3 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?