Missed It By That Much

In the mission to gain control over data chaos, a project is launched in order to implement a new system to help remediate the poor data quality that is negatively impacting decision-critical enterprise information.

The project appears to be well planned. Business requirements were well documented. A data quality assessment was performed to gain an understanding of the data challenges that would be faced during development and testing. Detailed architectural and functional specifications were written to guide these efforts.

The project appears to be progressing well. Business, technical and data issues all come up from time to time. Meetings are held to prioritize the issues and determine their impact. Some issues require immediate fixes, while other issues are deferred to the next phase of the project. All of these decisions are documented and well communicated to the end-user community.

Expectations appear to have been properly set for end-user acceptance testing.

As a best practice, the new system was designed to identify and report exceptions when they occur. The end-users agreed that an obsessive-compulsive quest to find and fix every data quality problem is a laudable pursuit but ultimately a self-defeating cause. Data quality problems can be very insidious and even the best data remediation process will still produce exceptions.

Although all of this is easy to accept in theory, it is notoriously difficult to accept in practice.

Once the end-users start reviewing the exceptions, their confidence in the new system drops rapidly. Even after some enhancements increase the number of records without an exception from 86% to 99% – the end-users continue to focus on the remaining 1% of the records that are still producing data quality exceptions.

Would you believe this incredibly common scenario can prevent acceptance of an overwhelmingly successful implementation?

How about if I quoted one of the many people who can help you get smarter than by only listening to me?

In his excellent book Why New Systems Fail: Theory and Practice Collide, Phil Simon explains:

“Systems are to be appreciated by their general effects, and not by particular exceptions…
Errors are actually helpful the vast majority of the time.”

In fact, because the new system was designed to identify and report errors when they occur:

“End-users could focus on the root causes of the problem and not have to wade through hundreds of thousands of records in an attempt to find the problem records.”

I have seen projects fail in the many ways described by detailed case studies in Phil Simon’s fantastic book. However, one of the most common and frustrating data quality failures is the project that was so close to being a success but the focus on exceptions resulted in the end-users telling us that we “missed it by that much.”

I am neither suggesting that end-users are unrealistic nor that exceptions should be ignored.

Reducing exceptions (i.e., poor data quality) is the whole point of the project and nobody understands the data better than the end-users. However, chasing perfection can undermine the best intentions.

In order to be successful, data quality projects must always be understood as an iterative process. Small incremental improvements will build momentum to larger success over time.

Instead of focusing on the exceptions – focus on the improvements.

And you will begin making steady progress toward improving your data quality.

And loving it!

The Data Quality Goldilocks Zone

Schrödinger’s Data Quality

The Nine Circles of Data Quality Hell

Link to original post