David Binkley: Data and the Reasonable Test

David Binkley, Senior Technical Account Manager, Harte-Hanks Trillium Software

In all of the projects that I’ve been involved with over the years, I can’t ever recall seeing a requirement from the customer specifying that the overall results comply with what I feel is the final data requirement – do the results pass the “reasonable” test.  Sure there have been plenty of projects where the customer has come up with target percentages or variance levels for the test cases but rarely do I recall a customer specifically asking to check if the end results appear reasonable. 

The reason that I’m bringing this up is that I have seen plenty of projects where we’ve been called in late in the implementation phase or shortly after going live where the customer is in a panic and making statements to the effect that “these results don’t appear reasonable”.  So is this panic a byproduct of not setting proper expectations or is it because the reality of the data’s quality has changed those expectations?  I venture to say that, in most cases, it’s likely been due to both.

Most of you will agree that it’s a matter of fact that one of the first goals for data quality is to provide the functionality of “due diligence” for your projects and applications and to be able to effectively perform that due diligence, you have to have at least an idea of what are reasonable results to begin with.  Defining reasonable can be tricky but you typically start off by asking questions about what facts are known about the data and what factors made the customer come to their assumptions and conclusions about what is right and wrong with it.  After gathering those assumptions, facts and conclusions and by doing some analysis of the unknowns, you can then have a better perspective from which you are able to formulate some initial expectations.

But that’s only the start of the exercise.  You need to remember that those initial reasonable expectations are just that – expectations.  Expectations don’t become reality until we perform our due diligence processes to validate the assumptions, gather the facts and implement the remediation processes to fulfill them.  We have to learn to trust what the customer is telling us is true, but we also need to verify as much as possible or we may be setting ourselves up for some sort of failure somewhere in the future.  Performing those due diligence processes will help you avoid the situations when the live data doesn’t support the business case or perhaps supports it too much – like suddenly surprising management with the fact that 40% of the customers are duplicates when expectations were that only 5% were duplicates. 

After you have performed your due diligence processes, I advocate that “the reasonable test” will also need to be applied against the end results.  No matter what the due diligence results were, you always need to ask yourself: Do the results make sense?  Ask yourself questions like: Given the fact that all the test cases worked, is it still reasonable that 25% of your billing records don’t have any addresses? 

Granted results outside the reasonable expectations often do occur but if we’ve correctly done our due diligence, those results at least shouldn’t surprise you the day after you have gone live with your project.  You should have known them well in advance of the implementation date giving you time to inform management that they were coming and to make plans to deal with them.

So, make sure that you have some reasonable expectations to start with and don’t be afraid to question unreasonable results – what seems unreasonable may also be incorrect.  At worst case, by questioning them and gathering the facts, you’ll then be prepared to defend those results when they are likely to be questioned by management or the end users.
   
There’s one quick example for those people who still are questioning this test.  Do you know how Bernie Madoff’s Ponzi scheme was finally exposed?  A couple of people simply challenged the returns on investment as not being reasonable as compared with everyone else.  Imagine how much pain could have been avoided if someone had performed the reasonable test years ago.