Reference Domains Part III: Collecting Classifications
702 Views
This is part three in the series about reference domains (a.k.a. classifications). In the first part we looked at the nature of classifications. In part two, we discussed a recommended approach to designing the target structures in the EDW to accommodate them. Here we will look at the work of collecting and documenting reference domains.
The compilation of a master list of reference domains and their values straddles the analysis and design phases of development.
This is part three in the series about reference domains (a.k.a. classifications). In the first part we looked at the nature of classifications. In part two, we discussed a recommended approach to designing the target structures in the EDW to accommodate them. Here we will look at the work of collecting and documenting reference domains.
The compilation of a master list of reference domains and their values straddles the analysis and design phases of development. For external classifications, those captured from the source system, the work must be done in the analysis phase, while internal classifications, being dependent on the design of the target data structures, are collected during the design phase.
As mentioned in part one, the external reference domains may first be encountered as code values in a source table; perhaps a status of a bank account, or a categorization of a customer. If you’re lucky, the word “code”, “type”, or “indicator” will appear in the column name. But it’s equally possible that the name will not make it immediately apparent that this is a classification. The analyst will need to determine it from the documentation or from the values contained within the data. Cryptic or abbreviated codes can take the form of numbers or character-based strings. It is important for the analyst to read the clues in the table layout, the documentation and the data to identify potential classifications and then follow up with the data steward or external provider of the source data to find the descriptions that explain the codes.
The code descriptions may accompany the codes in the same source table. However, even if this is the case, selecting a distinct set of the codes and descriptions from the table may not return a full set of potential values, only those that are currently being used. For example, pulling a distinct set of status types from a bank account table may return the values “open” and “closed”, but miss the fact that sometimes accounts are briefly “pending”. For this reason, it’s very important to find either a master table in the source that contains the full set of possible codes and values, or the documentation where the full set is listed. In either case, it’s essential to examine the list of occurring values and compare it against the master. The master may be out of sync with the newest list of allowable values, or there may be aberrant values appearing in the data which will need to be dealt with.
In some cases, no descriptions will exist. Business users may know the meaning of the codes, without requiring the definition. In these cases, it is best practice to create the description and include it in the reference value table. Codes should have explicit descriptions wherever possible, to avoid miscommunication and potential errors. It is a simple matter to misinterpret a status code of “A” as “Active”, when it actually means “Archived”.
Internal reference domains are quite different, in that they are entirely in the control of the EDW design team. The codes and descriptions are all generated and maintained internally. They take the same form of the external classifications, being codes and descriptions, and need to be compiled together with them in a master document and stored in the same reference tables in the EDW.
For compiling the master list, I recommend a pair of spreadsheets, one with a list of the reference domains (e.g., Account Type, Marital Status etc.) and the other a list of reference values for each domain. Here are some points to consider as you create and maintain this list:
- Manually set the surrogate keys
Rather than simply pass the setting of surrogate keys over to the ETL process, this master list is an opportunity to control the keys being applied. In most cases, each domain will contain a relatively short list of values (<50) and this will be a simple task. The control and predictability it supplies can be useful for development, testing and usage of the database. - Reserve ranges of values for the surrogate keys
It’s a good idea to leave room between the domains of values to allow the reference domain to grow or be corrected over time. These ranges will not be uniform; one domain will reserve twenty sequential numbers while another will require a thousand. - Create dummy values for “Unknowns”
Many times, incoming records will not find a match to the allowable domain of values. Rather than leave these values null, a surrogate key corresponding to an “unknown” reference value within the domain can be applied. - Create dummy values at finer grain
In some cases it may be prudent to distinguish between different types of “unknown” values. For example, records for which a classification is not required could be marked “Not Applicable”, while an incoming value that is out of the allowable range could be assigned “Out of Range”. When applying this finer grain consideration should be given to what is practical as well as what will add value. - Create load-ready spreadsheet
Some external classifications can be loaded from tables, while others, along with all the internal classifications will need to be manually entered. In both cases, a spreadsheet can be used to control the process. I have seen an implementation where a single spreadsheet contained both manual entries and references to tables, and was used as input to the ETL load process.
In the final part of this series we will look at the collection of metadata related to reference domains and how it can be used to assist the governance of data content.
Trending Now
1
Big Data
39 shares2,540 views
39 shares2,540 views
Types Of eCommerce Data You Should Note During Data Migration
Ryan Kh - September 4, 20192
Big Data
21 shares1,252 views
21 shares1,252 views
What To Learn About 4K Video Compression In The Age of Big Data
Diana Hope - September 2, 2019
4
Big Data
29 shares2,972 views
29 shares2,972 views
How Big Data Leads To Improvements In Company Letterhead Designs
Diana Hope - August 29, 20195
Big Data
39 shares2,941 views
39 shares2,941 views
Navigating The Big Data ICT Training Process In The UK
Sean Mallon - August 29, 20196
Big Data
45 shares3,615 views
45 shares3,615 views
6 Data-Driven Marketing Strategies That Are Revolutionizing Sales
Sean Parker - August 29, 2019
Hot Posts
1
Artificial Intelligence
42 shares2,152 views
42 2,152
Why Local SEO Strategies Require AI Technology – Here’s Why
Sean Parker - September 17, 20192
Machine Learning
23 shares2,118 views
23 2,118
5 Benefits Of Machine Learning In Enterprise Electrical Systems
Ryan Kh - September 17, 20194
Big Data
41 shares2,316 views
41 2,316
How Up-And-Coming Music Companies Use Big Data For Optimal Results
Annie Qureshi - September 15, 2019Latest
Why Local SEO Strategies Require AI Technology – Here’s Why
September 17, 2019
5 Benefits Of Machine Learning In Enterprise Electrical Systems
September 17, 2019
Big Data Paves The Road For A New Generation Of Investing Apps
September 15, 2019Popular
Data Visualization: A Scientific Treatment
November 7, 2014
User Adoption – Resistance Is Futile, We Hope
January 10, 2012
You must log in to post a comment.