We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
    From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
    12 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Automate Data Remediation to Find Dirty Data Before Your Customers Do
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Automate Data Remediation to Find Dirty Data Before Your Customers Do
Data QualityData WarehousingInside Companies

Automate Data Remediation to Find Dirty Data Before Your Customers Do

RickSherman
RickSherman
5 Min Read
Automate Data Remediation to Find Dirty Data Before Your Customers Do
Photo by mountpolley on Pixabay (https://pixabay.com/photos/mount-polley-hazeltine-creek-4770069/)
SHARE

I teach a graduate course on data warehousing at Northeastern University in Boston. Unlike the people I teach at clients’ sites or at conferences such as TDWI, most of my students have not actually worked in IT yet, never mind had hands-on experience with data warehousing and business intelligence.

This means I often have to go back to the basics. If I mention something like data remediation or rework, I’m sure to be asked what it is, why it matters, what causes it and what it has to do with enterprise data management (EDM).

What is data remediation?

The “what it is” is the easy part: business needs accurate information and that often requires going back to rework and fix data to eliminate data-quality issues. Data needs be checked for completeness, conformity, consistency, duplicates, integrity, and accuracy.

More Read

How MapR’s M7 Platform Improves NoSQL and Hadoop
How MapR’s M7 Platform Improves NoSQL and Hadoop
First Look: Via Science
Turning Data Into Content Through Social Media
The Tooth Fairy of Data Quality
The Importance of Cleaning Up Your Dark Data

A 2007 survey of more than 1,000 middle managers of large companies in the United States and United Kingdom conducted by Accenture revealed that “Middle managers spend more than a quarter of their time searching for information necessary to their jobs, and when they do find it, it is often wrong.” (The emphasis is mine.) What do you want to bet that this hasn’t changed since then?

Why is data remediation important?

The “why it matters” is tied into all the reasons why data quality matters. For my grad school students, I’d compare it to Six Sigma, something they’ve likely encountered in their management classes. The earlier in a process that you find defects the easier and less disruptive it is to fix them. YOU want to be the one to find the problems, not your customer. With auto manufacturing, defects found in design or manufacturing can be handled internally. But defects found by the customer mean expensive, embarrassing recalls that end up on the evening news. And even worse for the food or pharmaceutical industry, defects can be deadly to their customers.

It’s the same with data – if no one internally identifies quality problems and they then are found by the customer, you find yourself faced with a fire drill to fix them. You’re scrambling, your CIO is livid, and your image is blown along with the reputation of your CPM, BI or DW program.

That’s expensive and embarrassing, but what about the bad data no one finds that ends up being used to make business decisions? That could be damaging to your business.

External pressures like government regulations can put a lot of pressure on finance departments to start data remediation projects. The business risks of not doing it include all the problems associated with poor data quality. But if they try to do it manually (which many do, surprisingly) they’re apt to miss a lot. It takes longer, requires more time, and introduces complexities. In the scheme of EDM, data remediation should always be automated (e.g., with an ETL product) to improve the accuracy and overall business operations.

Enterprise Data Management (EDM) is the proactive approach to ensuring data is transformed into consistent, accurate and timely business information. And if there are data issues, you can discover them with auditing and data lineage rather than scrambling through macros in spreadsheets. A patch-work of stopgap measures is costly and you are not likely to solve data quality issues in a reactive manner, i.e. data remediation.

Let’s face it; you can’t fix a problem that you can’t find. And, it’s worth noting, that with data quality it’s finding the problems is paramount. You might not have to fix everything you find, but you sure better find it!

Data quality isn’t so much of a problem when you know where your problems lie. The time has come for both business and IT to address these issues through an EDM program.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article connects downstream report failures, data-quality gaps, and conflicting metric definitio
Signs Your Data Architecture Needs an Outside Review
Big Data Exclusive
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Big Data Exclusive
Flat editorial illustration: The article contrasts tools for gathering fresh interviews with tools for analyzing existing qualita
10 AI Market Research Tools That Turn Qualitative Data into Insights
Artificial Intelligence Big Data Exclusive
Flat editorial illustration: The article compares proxy MCP servers by what an AI agent can do through them: manage proxy account
Proxy MCP Servers: 7 Options and What Each Lets an AI Agent Do
Artificial Intelligence Big Data Data Collection

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Resource Mistakes, Part IV: Expecting Complete Obedience
Inside Companies

Resource Mistakes, Part IV: Expecting Complete Obedience

5 Min Read
3 Massive Cost-Saving Benefits of Smart Data for Businesses
Big Data

3 Massive Cost-Saving Benefits of Smart Data for Businesses

4 Min Read
What Really Is Big Data? And Why It Will Change the World
AnalyticsBig DataData MiningData WarehousingExclusiveModelingPredictive AnalyticsSentiment AnalyticsSocial Media AnalyticsText AnalyticsUnstructured DataWeb AnalyticsWorkforce Analytics

What Really Is Big Data? And Why It Will Change the World

13 Min Read
Gartner's 2009Q1 Magic Quadrant for BI Platforms
Business IntelligenceData Warehousing

Gartner’s 2009Q1 Magic Quadrant for BI Platforms

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
ai chatbot
How AI Website Chatbots Improve Customer Support and Lead Generation
Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?