Data Integration: Hand-coding Using ETL Tools

We are creatures of habit. It’s not easy to stop doing something the way we’ve always done it. Especially when we think we are right (but actually we’re not). Let’s explain.

I have discussed (some might say preached) in many posts, articles, webinars, podcasts, classes and client discussions that for any recurring data integration tasks IT should use an Extract, Transform and Load (ETL)  tool.

This certainly has been the best practice for enterprise data warehousing projects in the Fortune 1000. This is where I got my early experience in data integration and got to use the ETL tools that annually rank in Gartner’s Upper Magic Quadrant and Forrester’s Top Wave. These ETL tools enabled IT groups and SI (system integrator) project teams to tackle data integration challenges too complex and extensive for hand-coding. 

However, while the enterprise data warehousing projects were being developed with enterprise class ETL tools, most Fortune 1000 departmental projects and small to medium business (SMB) companies were hand-coding their data-integration processes.

IT groups choose to hand-code because for quite a while the enterprise class ETL tools they heard of were too …

We are creatures of habit. It’s not easy to stop doing something the way we’ve always done it. Especially when we think we are right (but actually we’re not). Let’s explain.

I have discussed (some might say preached) in many posts, articles, webinars, podcasts, classes and client discussions that for any recurring data integration tasks IT should use an Extract, Transform and Load (ETL)  tool.

This certainly has been the best practice for enterprise data warehousing projects in the Fortune 1000. This is where I got my early experience in data integration and got to use the ETL tools that annually rank in Gartner’s Upper Magic Quadrant and Forrester’s Top Wave. These ETL tools enabled IT groups and SI (system integrator) project teams to tackle data integration challenges too complex and extensive for hand-coding. 

However, while the enterprise data warehousing projects were being developed with enterprise class ETL tools, most Fortune 1000 departmental projects and small to medium business (SMB) companies were hand-coding their data-integration processes.

IT groups choose to hand-code because for quite a while the enterprise class ETL tools they heard of were too expensive for their budgets. In addition, these tools required dedicated, trained developers. Life in a small IT group means doing multiple tasks and never really having time to get training. The result is millions of lines of hand-coded ETL in enterprises today —  most of which is not documented and took much longer to develop than it would have with an ETL tool.

Times are Changing

But times have changed. There are now very robust, affordable ETL tools that can easily handle any departmental or SMB data integration projects (and might even be able to handle some enterprise data warehouses too).

In fact, two classes of ETL tools are free (or almost so). First, database vendors have “bundled” ETL tools with their databases. Although initially these tools were very simplistic they have expanded over the years have quite robust functionality. Second, open source software (OSS) ETL tools have also emerged and are capable of handling many departmental and SMB needs.

The emergence of these tools has broken through the pricing barrier with ETL tools that are not only quite capable, but not difficult for a small IT staff to learn. I have been advocating these tools in the departmental and SMB data integration market for years. The good news is these tools are picking up converts but the bad news is IT may not be using these tools as well as they might.

Stay tuned for my next post, where I continue this discussion with Hand-Coding: What Went Wrong and How to Avoid Repeating History’s Mistakes

Link to original post