Sitting in any data quality competency centre, or working on any data warehousing support team, there are times when you come across data that just screams at you, and not in a good way. (Kind of like the ghost from the Grudge.) The data’s gone through all data processing without a hitch, business says it’s good data, but something tells you it’s just not right. That my friends is called intuition.
My recommendation would be to use it.
Intuition does not come naturally, it comes to you over time. After you’ve gained an understanding of the processes and learned about the business, intuition will set you above the other data analysts.
Business Knowledge
Know the business, I’ve mentioned this before in my post about Data Quality Analyst Attributes.
What do you do when you have 23,000 service cancellations on one order. This would be red flagged immediately in most service oriented companies. It might even cause a job to stop processing.
The process tells you there’s a problem. The business would rightfully tell you to investigate, it can’t be right. However, someone in the business knows what the truth is. Check the data, you just may find the indicator to identify the type of customer it is, and easily learn that the business is justified in cancelling 23,000 service items. Perhaps the business lost a call centre customer.
Process Knowledge
Processes will establish checks and validation points that tell you when the data is good and bad. It will tell you where the data is and what happens to it at specific system touch points and more. You must know this in order to understand the process and the data that goes through the processes. Having a good understanding of the processes will allow you to identify where and when errors could occur with your data sets. Process knowledge will guide you to where you need to correct the data or processes.
Intuition
Your knowledge, your understanding of the data will guide you and your intuitive feeling to determine what is right. There is no substitute for it. Intuition will give you that sixth sense and you will be able to differentiate true data issues from false data issues when the tools and processes set in place cannot make that differentiation. Use it when the tools just don’t cut it.
Example:
The process says the CRM application must match the records, the business believes the records should be matched and even want them to match. You know that even if John Smith, and John Smith living in the same city aren’t necessarily the same person, two positives matches does not necessarily make a single profile.
