By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data Analytics instagram stories
    Data Analytics Helps Marketers Make the Most of Instagram Stories
    15 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    What to Know Before Recruiting an Analyst to Handle Company Data
    6 Min Read
    AI analytics
    AI-Based Analytics Are Changing the Future of Credit Cards
    6 Min Read
    data overload showing data analytics
    How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
    8 Min Read
    hire a marketing agency with a background in data analytics
    5 Reasons to Hire a Marketing Agency that Knows Data Analytics
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Be Prepared to Duel with Data Quality
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Be Prepared to Duel with Data Quality
Business IntelligenceData QualityData Warehousing

Be Prepared to Duel with Data Quality

RickSherman
Last updated: 2011/08/10 at 12:49 PM
RickSherman
10 Min Read
SHARE

Word_data-quality007 Plenty of business intelligence (BI) or data warehouse projects have been blindsided by complications related to data quality.

Word_data-quality007 Plenty of business intelligence (BI) or data warehouse projects have been blindsided by complications related to data quality. Sometimes these issues aren’t apparent until business users start testing the systems just before going live with the projects. What causes BI project teams to get caught off guard by data quality issues? Why do these problems surface so late in the projects?

There are two common pitfalls: defining data quality too narrowly and assuming data quality is the responsibility of the source systems.

People often assume that data quality simply means eliminating bad data – data that is missing, inaccurate or incorrect. Bad data is certainly a problem, but it isn’t the only problem. Good data quality programs also ensure that data is comprehensive, consistent, relevant and timely.

More Read

ai low code frameworks

AI Can Help Accelerate Development with Low-Code Frameworks

Tackling Bias in AI Translation: A Data Perspective
How AI is Boosting the Customer Support Game
AI-Based Analytics Are Changing the Future of Credit Cards
Enterprises Are Leveraging the Benefits of AI-Driven ERPs
Don’t Blame the Source Systems
Defining data quality too narrowly often leads people to assume that source transactional systems – either through data entry or systemic errors – cause the bad data. Although they may be a source of some errors, the more likely culprits are either inconsistent dimensions across source systems (such as customer or product identifiers) or inconsistent definitions for derived data across organizations. Conforming dimensions – developing consistent customer or product identifiers – is important for accessing and analyzing data for a company. The source systems do not own the data quality issues across other systems- the BI project team does. The source systems need to ensure that the data within their data silos is correct. However, the BI project team is responsible for providing the business with data that is consistent across the enterprise.
Similarly, each organization within the enterprise may have valid business reasons to derive data differently than others. For example, their position in a set of business processes may determine how they view their data. The individual organizations aren’t tasked with developing common definitions for derived data, but the BI project team is. Many BI project teams try to claim that data quality issues aren’t their responsibility. However, from a practical viewpoint, the BI team does need to make these issues its own, because its job is to ensure the highest data quality possible. The BI project team is packaging the data for consumption by business users, and they will be held accountable for the data quality. This may not seem fair, but the success of their project depends on it.
Don’t Shortchange the Pilot
Surprises happen when the project does an initial pilot or release involving only a small subset of source systems. While there may be many good reasons to have a narrow scope for a pilot, you won’t get an appreciation for the effort necessary to conform these dimensions as the number of source systems expands.
Sometimes pilots are only with a single organization, using only their definitions for derived data. Once again, the tough issue is often how to accommodate the differences in the derivation definitions between organizations. In both cases, the real challenges are encountered when dealing with multiple systems and organizations. The business users need to look at the big picture, and that is only possible when they can access and analyze data across the enterprise.
Steps to Address Data Quality
To ensure data quality, the BI project team needs to address it from the very beginning. Here are several significant steps to consider:
Require the business to define data quality in a broad sense, establish metrics to monitor and measure it, and determine what should be done if the data fails to meet these metrics.
Undertake a comprehensive data profiling effort when performing a source systems analysis. Data anomalies across source systems and time (historical data does not always age well!) are needed so that the team can address them with the business early.
Incorporate data quality into all data integration and business intelligence processes from data sourcing to information consumption by the business user. Data quality issues need to be detected as early in the processes as possible and dealt with as defined in the business requirements.
Enterprises must present data that meets very stringent data quality levels, especially in light of recent compliance regulations and demands. The level of data transparency needed can only result from establishing a strong commitment to data quality and building the processes to ensure it.

Don’t Blame the Source Systems

Defining data quality too narrowly often leads people to assume that source transactional systems – either through data entry or systemic errors – cause the bad data. Although they may be a source of some errors, the more likely culprits are either inconsistent dimensions across source systems (such as customer or product identifiers) or inconsistent definitions for derived data across organizations. Conforming dimensions – developing consistent customer or product identifiers – is important for accessing and analyzing data for a company. The source systems do not own the data quality issues across other systems- the BI project team does. The source systems need to ensure that the data within their data silos is correct. However, the BI project team is responsible for providing the business with data that is consistent across the enterprise.

Similarly, each organization within the enterprise may have valid business reasons to derive data differently than others. For example, their position in a set of business processes may determine how they view their data. The individual organizations aren’t tasked with developing common definitions for derived data, but the BI project team is. Many BI project teams try to claim that data quality issues aren’t their responsibility. However, from a practical viewpoint, the BI team does need to make these issues its own, because its job is to ensure the highest data quality possible. The BI project team is packaging the data for consumption by business users, and they will be held accountable for the data quality. This may not seem fair, but the success of their project depends on it.

Don’t Shortchange the Pilot

Surprises happen when the project does an initial pilot or release involving only a small subset of source systems. While there may be many good reasons to have a narrow scope for a pilot, you won’t get an appreciation for the effort necessary to conform these dimensions as the number of source systems expands.
Sometimes pilots are only with a single organization, using only their definitions for derived data. Once again, the tough issue is often how to accommodate the differences in the derivation definitions between organizations. In both cases, the real challenges are encountered when dealing with multiple systems and organizations. The business users need to look at the big picture, and that is only possible when they can access and analyze data across the enterprise.Steps to Address Data Quality

To ensure data quality, the BI project team needs to address it from the very beginning. Here are several significant steps to consider:

  • Require the business to define data quality in a broad sense, establish metrics to monitor and measure it, and determine what should be done if the data fails to meet these metrics.
  • Undertake a comprehensive data profiling effort when performing a source systems analysis. Data anomalies across source systems and time (historical data does not always age well!) are needed so that the team can address them with the business early.
  • Incorporate data quality into all data integration and business intelligence processes from data sourcing to information consumption by the business user. Data quality issues need to be detected as early in the processes as possible and dealt with as defined in the business requirements.

Enterprises must present data that meets very stringent data quality levels, especially in light of recent compliance regulations and demands. The level of data transparency needed can only result from establishing a strong commitment to data quality and building the processes to ensure it.

RickSherman August 10, 2011
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

smart home data
7 Mind-Blowing Ways Smart Homes Use Data to Save Your Money
Big Data
ai low code frameworks
AI Can Help Accelerate Development with Low-Code Frameworks
Artificial Intelligence
data Analytics instagram stories
Data Analytics Helps Marketers Make the Most of Instagram Stories
Analytics
data breaches
How Hospital Security Breaches Devastate Local Communities
Policy and Governance

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

ai low code frameworks
Artificial Intelligence

AI Can Help Accelerate Development with Low-Code Frameworks

12 Min Read
data perspective
Big Data

Tackling Bias in AI Translation: A Data Perspective

9 Min Read
How AI is Boosting the Customer Support Game
Artificial Intelligence

How AI is Boosting the Customer Support Game

6 Min Read
AI analytics
AnalyticsArtificial IntelligenceExclusive

AI-Based Analytics Are Changing the Future of Credit Cards

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?