I’m not only concerned about hand-coding versus ETL tools; I’m also concerned that potential buyers of ETL tools and the market in general are only looking at a small number of players in the ETL marketplace.
For many years industry analyst research groups have identified the top two product vendors: Informatica and IBM (from its acquisition of Ascential Software). So, naturally, these two appear on any evaluation shortlist. The rest of the evaluation shortlist usually includes the bundled products (mentioned in my recent posts) that come with the databases, BI tools or applications that the evaluating company already owns. Beyond these usual suspects, other ETL or data integration products are pretty obscure and almost invisible, at least from a general market perspective.
There are many ETL and integration products in the marketplace that lie somewhere between the mega-products and the bundled tools in terms of functionality and total cost of ownership (TCO). The problem for the vendors that offer these products is that very few people know about them — other than data integration nerds like me and the vendors’ relatives!
To make matters even worse, industry analysts also seem to ignore these vendors because of their company size and relatively small installed base. I am not arguing with the industry analyst firms’ objective criteria to include/exclude vendors, but it definitely works against the “unknown” vendors.
Also, to add insult to injury, industry pundits generally speak and write about the high-tech titans and ignore the smaller vendors. If everyone only talks about the well-known vendors, then the smaller vendors will remain unknown.
Conventional wisdom says…
If you need it and can afford it a data integration project will be successful if you buy the “best-in-class” tools. The old IT industry adage was you can’t get fired for buying IBM. Let’s expand that in the current world of ETL and say you can’t get fired if you buy IBM or Informatica (although you can if you don’t complete the project on time and on budget… and in this economy no one’s job is completely safe.)
On the other end of the spectrum, where budgets and data integration needs are limited, the bundled tools may indeed do the job. My qualifier, as I mentioned in my previous post, is that if you skip learning the what and how of data integration processing then you could easily fail at using these tools.
Change the paradigm…
Completing the data integration project tasks is fine, but what about TCO, skills needed and time to market? The answer is that there are other ETL products in the marketplace that compete with the Titans and bundled products. Broaden your evaluation shortlist beyond the usual suspects. Maybe you will find a better fit for your data integration tasks than the “best-in-class” or the cheapest option. Or maybe you’ll still choose the usual suspects. But either way, at least you will have taken a good look at the possibilities.
Final thoughts
Advice to ETL evaluators: broaden your search and you might be pleasantly surprised.
Advice to ETL vendors: get visible. First, get recognized. Second, differentiate yourself. Third, your message needs to be understandable to people who do not know ETL.
