DQ is 1/3 Process Knowledge + 1/3 Business Knowledge + 1/3 Intuition

May 26, 2009
48 Views

Sitting in any data quality competency centre, or working on any data warehousing support team, there are times when you come across data that just screams at you, and not in a good way. (Kind of like the ghost from the Grudge.) The data’s gone through all data processing without a hitch, business says it’s good data, but something tells you it’s just not right. That my friends is called intuition.

My recommendation would be to use it.

Intuition does not come naturally, it comes to you over time. After you’ve gained an understanding of the processes and learned about the business, intuition will set you above the other data analysts.

Business Knowledge
Know the business, I’ve mentioned this before in my post about Data Quality Analyst Attributes.

What do you do when you have 23,000 service cancellations on one order. This would be red flagged immediately in most service oriented companies. It might even cause a job to stop processing.

The process tells you there’s a problem. The business would…


Sitting in any data quality competency centre, or working on any data warehousing support team, there are times when you come across data that just screams at you, and not in a good way. (Kind of like the ghost from the Grudge.) The data’s gone through all data processing without a hitch, business says it’s good data, but something tells you it’s just not right. That my friends is called intuition.

My recommendation would be to use it.

Intuition does not come naturally, it comes to you over time. After you’ve gained an understanding of the processes and learned about the business, intuition will set you above the other data analysts.

Business Knowledge
Know the business, I’ve mentioned this before in my post about Data Quality Analyst Attributes.

What do you do when you have 23,000 service cancellations on one order. This would be red flagged immediately in most service oriented companies. It might even cause a job to stop processing.

The process tells you there’s a problem. The business would rightfully tell you to investigate, it can’t be right. However, someone in the business knows what the truth is. Check the data, you just may find the indicator to identify the type of customer it is, and easily learn that the business is justified in cancelling 23,000 service items. Perhaps the business lost a call centre customer.

Process Knowledge
Processes will establish checks and validation points that tell you when the data is good and bad. It will tell you where the data is and what happens to it at specific system touch points and more. You must know this in order to understand the process and the data that goes through the processes. Having a good understanding of the processes will allow you to identify where and when errors could occur with your data sets. Process knowledge will guide you to where you need to correct the data or processes.

Intuition
Your knowledge, your understanding of the data will guide you and your intuitive feeling to determine what is right. There is no substitute for it. Intuition will give you that sixth sense and you will be able to differentiate true data issues from false data issues when the tools and processes set in place cannot make that differentiation. Use it when the tools just don’t cut it.

Example:
The process says the CRM application must match the records, the business believes the records should be matched and even want them to match. You know that even if John Smith, and John Smith living in the same city aren’t necessarily the same person, two positives matches does not necessarily make a single profile.

You may be interested

How SAP Hana is Driving Big Data Startups
Big Data
298 shares3,139 views
Big Data
298 shares3,139 views

How SAP Hana is Driving Big Data Startups

Ryan Kh - July 20, 2017

The first version of SAP Hana was released in 2010, before Hadoop and other big data extraction tools were introduced.…

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion
Data Management
120 views
Data Management
120 views

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion

Manish Bhickta - July 20, 2017

Physical Data destruction techniques are efficient enough to destroy data, but they can never be considered eco-friendly. On the other…

10 Simple Rules for Creating a Good Data Management Plan
Data Management
69 shares689 views
Data Management
69 shares689 views

10 Simple Rules for Creating a Good Data Management Plan

GloriaKopp - July 20, 2017

Part of business planning is arranging how data will be used in the development of a project. This is why…