The data integration process is traditionally thought of in three steps: extract, transform, and load (ETL). Putting aside the often-discussed order of their execution, “extract” is pulling data out of a source system, “transform” means validating the source data and converting it to the desired standard (e.g. yards to meters), and load means storing the data at the destination.
The data integration process is traditionally thought of in three steps: extract, transform, and load (ETL). Putting aside the often-discussed order of their execution, “extract” is pulling data out of a source system, “transform” means validating the source data and converting it to the desired standard (e.g. yards to meters), and load means storing the data at the destination.
An additional step, data “enrichment”, has recently emerged, offering significant improvement in business value of integrated data. Applying it effectively requires a foundation of sound data management practices.
Data integrators traditionally bring data from source to target unchanged. It’s as if ETL developers were movers who prided themselves on putting your furniture in the new place unbroken. Businesses today are asking the movers to repair and improve the furniture before landing it in the new house.
The most obvious enrichment example is address correction. When you enter your address on some US e-commerce sites, the site corrects it by standardizing street, city, and state fields, and adding the last four digits of the zip code. ETL vendors tout many possibilities beyond address correction. One lists these types of information that can be added, or “augmented“, to a demographics database, presumably from databases that vendor can provide:
- “Geographic: such as post code, county name, longitude and latitude, and political district
- “Behavioral: including purchases, credit risk and preferred communication channels
- “Demographic: such as income, marital status, education, age and number of children
- “Psychographic: ranging from hobbies and interests to political affiliation
- “Census: household and community data”
Enrichment isn’t limited to demographics. Data quality tools like this one allow definition of rules that integrate into the ETL stream for any data source:
- Matching incoming records with existing data, like identifying to which insured member a claim applies
- Correcting invalid data based on other data in the record, like correcting an out-of-bounds hand-entered measurement based on an independent automated data feed
- Interpolating missing values based on other available data. So while loading a pregnancy related claim the system might fill in a missing value for gender.
As you can imagine, changing source data runs counter to most integrators’ instincts. And yes, it’s risky. Operations that automatically match, correct, or interpolate data values operate with some “confidence” level, meaning that sometimes they are wrong. I worked in one customer service organization whose matching routines processed tens of millions of customer records with 95% confidence. That meant that hundreds of thousands of matches may have been incorrect — not necessarily an issue for the particular application involved, but something for those implementing enrichment to consider.
Given those risks, I suggest these three guiding principles for organizations adding enrichment to their data integration streams:
- The business should drive and manage enrichment definition: Data stewards who understand the incoming data and the intended use must be the key drivers of what data is enriched, how it is done, and test of the enrichment outcomes.
- Enriched data must be identifiable and audit-able in the target database: Any integration target database should feature complete lineage metadata: where is this data element from, when was it loaded, and what happened to it along the way. This is even more true for data added by interpolating from, augmenting, matching, or correcting source data. Analysts must know which data came directly from the source, which was generated, and the confidence level of the latter.
- Data replaced by enrichment must be available alongside the enriched data: Enrichment processes must store modified or added data in such a way that analysts have access to the “raw” source data. Analysts should be able to independently test enrichment processes and suggest improvements if needed. If, for whatever reason, enrichment doesn’t meet specific analysis needs, then they should be able to fall back to the original source data.
By following these three guiding principles, organizations can ensure that they deploy enrichment processes that enhance business value of integrated data while minimizing risk and maximizing flexibility as requirements evolve.
