Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    sales and data analytics
    How Data Analytics Improves Lead Management and Sales Results
    9 Min Read
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Fudging Data: The How, The Why and Catching it
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Fudging Data: The How, The Why and Catching it
Data Mining

Fudging Data: The How, The Why and Catching it

Editor SDC
Editor SDC
5 Min Read
SHARE

An often encountered problem in data management as well as reporting is data inaccuracy. I was tempted to write about this while poring through reams of data that specifically I had been told to investigate for veracity. Why data is fudged- Some of data problems are  due to bad data gathering systems, some of it are […]


An often encountered problem in data management as well as reporting is data inaccuracy. I was tempted to write about this while poring through reams of data that specifically I had been told to investigate for veracity.

Why data is fudged–

More Read

Netflix zip-code data: How about political overlay?
Workplace Ups and Downs
Big Data and the Call for Evidence-based Management
Using Google Docs for Web Scraping
Is Your Company Ready to Deploy Business Intelligence Intelligently?

Some of data problems are  due to bad data gathering systems, some of it are due to wrong specifications, and some of it is often plain bad or simplistic assumptions.

Data fudging on the other hand is clearly inventing data to fit the curve or trend, and is deliberate and thus harder to catch.

It can also be included to give confusing rather than inaccurate data just to avoid greater scrutiny.

Sometimes it may be termed as over-fitting but over-fitting is generally due to statistical and programmatic reasons rather than human reasons.

 

Note fudging data or talking about is not really political correct in the data world , yet it exists all all levels from students preparing survey samples to budgetary requests.

I am outlining some ways in how to recognize data fudging – and to catch a fudge, you sometime have to think like one.

How data is often fudged-

  1. Factors-This starts be recognizing all factors that can positively or negatively impact the final numbers that are being presented.Note the list can be expanded to many more factors than needed just to divert attention from main causal factors.
  2. Sensitivity-This gives the range of answers gotten by tweaking individual factors within a certain range say +- 10 % and noting the final figures.Assumptions can be both conservative or aggressive in  terms of recognizing the weightage of causal factors in order to suit the final numbers.
  3. Causal Equation-Recognizing the interplay between various factors due to correlation as well to the final numbers due to causing variance changes.The causal equation can then be tweaked including playing with weightage, powers of polynomial expression, as well correlation between many factors.

How data fudging is often caught-

  1. Sampling- Using a random sample or holdout sample, and thus seeking if final answer converges to that known to happen. The validation sample technique is powerful to recognize data modeling inaccuracies.
  2. Checking assumptions- For reasons of risk management, always consider conservative or worst case scenarios first and then build up your analysis. Similarly for checking an analysis , check for over optimism or the period or history on which the assumption growth factors/sensitivities are assumed.
  3. Missing Value and Central Value Movements- If a portion of data is missing, check the mean as well as median for both the reported as well overall data. You can try and resample by taking a random sample from the data and check these values repeatedly to see if they hold firm.
  4. Early Warning Indicators-Ask the question (loudly)- if this analysis was totally wrong , what indicator would give us the first indication of it being wrong. This could be then incorporated as part of metric tracking early warning system

Note the above are simplistic expressions of numbers I have seen being presented wrongly, or being fudged. They are based on my experiences so feel free to add in your share of data anecdotes.

Using these simple techniques could have helped many people in the financial as well as other decision making including budgetary as well as even in other strategic areas.

As the saying goes- In God we Trust, Everybody else has to bring data ( which we will have to check before trusting it)

 

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sales and data analytics
How Data Analytics Improves Lead Management and Sales Results
Analytics Big Data Exclusive
ai in marketing
How AI and Smart Platforms Improve Email Marketing
Artificial Intelligence Exclusive Marketing
AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

Cyberlaw scholar Jonathan Zittrain of Harvard: Ubiquitous human…

1 Min Read

On ethical issues of randomized experiments

3 Min Read

How is Performance Management like Multi-wavelength Astronomy?

4 Min Read

100 Petabytes of Data in Poop?

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?