ETL code and processes are key in the operations of updating and maintaining a data warehouse. A good operational set-up will have data error reports available for the data quality analyst to verify and action.
Traditionally an error report will tell you the data is wrong, provide an error code which relates to a very lame description of the error. In many cases it was written by a business analyst who has worked a long-time on the project and just wants to wrap up the work and move on to the next project. Therefore, in those cases the support analyst or data analyst will be left holding the bag, or scratching their head when they do get a data error. Don’t let it get to that, push back and ask for more.
These are a few items to include in the error reports to make life easier for the support team, other than an error code.
Include:
1. Job number/name: this is an easy find to report on, this is the job that was handling the data when the error occurred.
2. Impacted Table: not everyone will know what job impacts what table, so include it, this will provide immediate recognition of the type of data your dealing with.
3. Primary Index: having the primary index will allow you to search f…
ETL code and processes are key in the operations of updating and maintaining a data warehouse. A good operational set-up will have data error reports available for the data quality analyst to verify and action.
Traditionally an error report will tell you the data is wrong, provide an error code which relates to a very lame description of the error. In many cases it was written by a business analyst who has worked a long-time on the project and just wants to wrap up the work and move on to the next project. Therefore, in those cases the support analyst or data analyst will be left holding the bag, or scratching their head when they do get a data error. Don’t let it get to that, push back and ask for more.
These are a few items to include in the error reports to make life easier for the support team, other than an error code.
Include:
1. Job number/name: this is an easy find to report on, this is the job that was handling the data when the error occurred.
2. Impacted Table: not everyone will know what job impacts what table, so include it, this will provide immediate recognition of the type of data your dealing with.
3. Primary Index: having the primary index will allow you to search for the record that is at fault either in the temporary table, or in a temporary holding file.
4. A brief meaningful description of the error, a two-word error message just does not cut it in many situations.
5. The most important piece of information to include in an error report would be to identify the action required to fix it. This is often excluded by many business analysts as they gather requirements. Data quality is often omitted from requirements and the business does not know the data the way they should (but are getting better), and have no idea how to correct it. It is a significant piece of information that will save significant amounts of time on the part of your support team. In the event you are not able to put the action required in an error report (due to possible text length or most often funding), reference the error back to the appropriate document that contains the actions required.
Just remember no one likes to get an error report, alert or notification identifying what has broken next. So make it easier for all those support analysts out there, give them what they need.