Who should be accountable for data quality?

The cardinality of a countable set - ex-mathematicians are allowed the occasional pun

		CIO Magazine forum

Asking the wrong question

Once more this post is inspired by a conversation on LinkedIn.com, this time the CIO Magazine forum and a thread entitled BI tool[s] can not deliver the expected results unless the company focuses on quality of data posted by Caroline Smith (normal caveat: you must be a member of LinkedIn.com and the group to view the actual thread).

The discussion included the predictable references to GIGO, but conversation then moved on to who has responsibility for data quality, IT or the business.

The cardinality of a countable set - ex-mathematicians are allowed the occasional pun

		CIO Magazine forum

Asking the wrong question

The discussion included the predictable references to GIGO, but conversation then moved on to who has responsibility for data quality, IT or the business.

As regular readers of this column will know, I view this as an unhelpful distinction. My belief is that IT is a type of business department, with specific skills, but engaged in business work and, in this, essentially no different to say the sales department or the strategy department. Looking at the question through this prism, it becomes tautological. However, if we ignore my peccadillo about this issue, we could instead ask whether responsibility for data quality should reside in IT or not-IT (I will manfully resist the temptation to write ~IT or indeed IT’); with such a change, I accept that this is now a reasonable question.

Answering a modified version of the question

In information technology, telecommunications, and related fields, handshaking is an automated process of negotiation that dynamically sets parameters of a communications channel established between two entities before normal communication over the channel begins. It follows the physical establishment of the channel and precedes normal information transfer.

My basic answer is that both groups will bring specific skills to the party and a partnership approach is the one that is most likely to end in success. There are however some strong arguments for IT playing a pivotal role and my aim is to expand on these in the rest of this article.

Before I enumerate these, one thing that I think is very important is that data quality is seen as a broad issue that requires a broad approach to remedy it. I laid out what I see as the four pillars of improving data quality in an earlier post: Using BI to drive improvements in data quality. This previous article goes into much more detail about the elements of a successful data quality improvement programme and its title provides a big clue as to what I see as the fourth pillar. More on this later.

1. The change management angle

Again, as with virtually all IT projects, the aim of a data quality initiative is to drive different behaviours. This means that change management skills are just as important in these types projects as in the business intelligence work that they complement. This is a factor to consider when taking decisions about who takes the lead in looking to improve data quality; who amongst the available resources have established and honed change management skills? The best IT departments will have a number of individuals who fit this bill, if not-IT has them as well, then the organisation is spoilt for choice.

2. The pan-organisational angle

Elsewhere I have argued that BI adds greatest value when it is all-pervasive. The same observations apply to data quality. If we assume that an organisation has a number of divisions, each with their own systems (due to the nature of their business and maybe also history), but also maybe sharing some enterprise applications. While it would undeniably be beneficial for Division A to get their customer files in order, it would be of even greater value if all divisions did this at the same time and with a consistent purpose. This would allow the dealings of Customer X across all parts of the business to be calculated and analysed. It could also drive cross-selling opportunities in particular market segments.

While it is likely that a number of corporate staff of different sorts will have a very good understanding about the high-level operations of each of the divisions, it is at least probable that only IT staff (specifically those engaged in collating detailed data from each division for BI purposes) will have an in-depth understanding of how transactions and master data are stored in different ways across the enterprise. This knowledge is a by-product of running a best practice BI project and the collateral intellectual property built up can be of substantial business value.

3. The BI angle

It was this area that formed the backbone of the earlier data quality article that I referenced above. My thesis was that you could turn the good data quality => good BI relationship on its head and use the BI tool to drive data quality improvements. The key here was not to sanitise data problems, but instead to expose them, also leveraging standard BI functionality like drill through to allow people to identify what was causing an issue.

One of the most pernicious data quality issues is of the valid, but wrong entry. For example a transaction is allocated a category code of X, which is valid, but the business event demands the value Y. Sometimes it is possible to guard against this eventuality by business rules, e.g. Product A can only be sold by Business Unit W, but this will not be possible for all such data. A variant of this issue is data being entered in the wrong field. Having spent a while in the Insurance industry, it was not atypical for a policy number to be entered as a claim value for example. Sometimes there is no easy systematic way to detect this type of occurrence, but exposing issues in a well-designed BI system is one way of noticing odd figures and then – crucially – being able to determine what is causing them.

4. The IT character angle

I was searching round for a way to put this nicely and then realised that Jim Harris had done the job for me in naming his excellent Obsessive-Compulsive Data Quality blog (OCDQ Blog). I’m an IT person, I may have general management experience and a reasonable understanding of many parts of business, but I remain essentially an IT person. Before that, I was a Mathematician. People in both of those lines of work tend to have a certain reputation; to put it positively, the ability to focus extremely hard on something for long periods is a common characteristic.

Aside: for the avoidance of doubt, as I pointed out in Pigeonholing – A tragedy, the fact that someone is good at the details does not necessarily preclude them from also excelling at seeing the big picture – in fact without a grasp on the details the danger of painting a Daliesque big picture is perhaps all to real!

Improving data quality is one of the areas where this personality trait pays dividends. I’m sure that there are some marketing people out there who have relentless attention to detail and whose middle name is “thoroughness”, however I suspect there are rather less of them than among the ranks of my IT colleagues. While leadership from the pertinent parts of not-IT is very important, a lot of the hard yards are going to be done by IT people; therefore it makes sense if they have a degree of accountability in this area.

In closing

Much like most business projects, improving data quality is going to require a cross-functional approach to achieve its goals. While you often hear the platitudinous statement that “the business must be responsible for the quality of its own data”, this ostensible truism hides the fact that one of the best ways for not-IT to improve the quality of an organisation’s data is to get IT heavily involved in all aspects of this work.

IT for its part can leverage both its role as one of the supra-business unit departments and its knowledge of how business transactions are recorded and move from one system to another to become an effective champion of data quality.