By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
    benefits of data analytics for financial industry
    Fascinating Changes Data Analytics Brings to Finance
    7 Min Read
    analyzing big data for its quality and value
    Use this Strategic Approach to Maximize Your Data’s Value
    6 Min Read
    data-driven seo for product pages
    6 Tips for Using Data Analytics for Product Page SEO
    11 Min Read
    big data analytics in business
    5 Ways to Utilize Data Analytics to Grow Your Business
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Data Integration: Hand-coding Using ETL Tools
Share
Notification Show More
Latest News
cloud-centric companies using network relocation
Cloud-Centric Companies Discover Benefits & Pitfalls of Network Relocation
Cloud Computing
construction analytics
5 Benefits of Analytics to Manage Commercial Construction
Analytics
database compliance guide
Four Strategies For Effective Database Compliance
Data Management
Digital Security From Weaponized AI
Fortifying Enterprise Digital Security Against Hackers Weaponizing AI
Security
DevOps on cloud
Optimizing Cost with DevOps on the Cloud
Cloud Computing Development Exclusive IT
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Warehousing > Data Integration: Hand-coding Using ETL Tools
Data Warehousing

Data Integration: Hand-coding Using ETL Tools

RickSherman
Last updated: 2009/11/04 at 7:23 PM
RickSherman
5 Min Read
SHARE
- Advertisement -

Hand-stop We are creatures of habit. It’s not easy to stop doing something the way we’ve always done it. Especially when we think we are right (but actually we’re not). Let’s explain.

I have discussed (some might say preached) in many posts, articles, webinars, podcasts, classes and client discussions that for any recurring data integration tasks IT should use an Extract, Transform and Load (ETL)  tool.

- Advertisement -

This certainly has been the best practice for enterprise data warehousing projects in the Fortune 1000. This is where I got my early experience in data integration and got to use the ETL tools that annually rank in Gartner’s Upper Magic Quadrant and Forrester’s Top Wave. These ETL tools enabled IT groups and SI (system integrator) project teams to tackle data integration challenges too complex and extensive for hand-coding. 

However, while the enterprise data warehousing projects were being developed with enterprise class ETL tools, most Fortune 1000 departmental projects and small to medium business (SMB) companies were hand-coding their data-integration processes.

More Read

What is Data Pipeline A detailed explaination

What is Data Pipeline? A Detailed Explanation

Understanding ETL Tools as a Data-Centric Organization
Differentiating Between Data Lakes and Data Warehouses
How Will The Cloud Impact Data Warehousing Technologies?
Big Data Is More Prevalent in Daily Life Than You Might Think

IT groups choose to hand-code because for quite a while the enterprise class ETL tools they heard of were too …



Hand-stop We are creatures of habit. It’s not easy to stop doing something the way we’ve always done it. Especially when we think we are right (but actually we’re not). Let’s explain.

I have discussed (some might say preached) in many posts, articles, webinars, podcasts, classes and client discussions that for any recurring data integration tasks IT should use an Extract, Transform and Load (ETL)  tool.

- Advertisement -

This certainly has been the best practice for enterprise data warehousing projects in the Fortune 1000. This is where I got my early experience in data integration and got to use the ETL tools that annually rank in Gartner’s Upper Magic Quadrant and Forrester’s Top Wave. These ETL tools enabled IT groups and SI (system integrator) project teams to tackle data integration challenges too complex and extensive for hand-coding. 

However, while the enterprise data warehousing projects were being developed with enterprise class ETL tools, most Fortune 1000 departmental projects and small to medium business (SMB) companies were hand-coding their data-integration processes.

IT groups choose to hand-code because for quite a while the enterprise class ETL tools they heard of were too expensive for their budgets. In addition, these tools required dedicated, trained developers. Life in a small IT group means doing multiple tasks and never really having time to get training. The result is millions of lines of hand-coded ETL in enterprises today —  most of which is not documented and took much longer to develop than it would have with an ETL tool.

Times are Changing

But times have changed. There are now very robust, affordable ETL tools that can easily handle any departmental or SMB data integration projects (and might even be able to handle some enterprise data warehouses too).

- Advertisement -

In fact, two classes of ETL tools are free (or almost so). First, database vendors have “bundled” ETL tools with their databases. Although initially these tools were very simplistic they have expanded over the years have quite robust functionality. Second, open source software (OSS) ETL tools have also emerged and are capable of handling many departmental and SMB needs.

The emergence of these tools has broken through the pricing barrier with ETL tools that are not only quite capable, but not difficult for a small IT staff to learn. I have been advocating these tools in the departmental and SMB data integration market for years. The good news is these tools are picking up converts but the bad news is IT may not be using these tools as well as they might.

Stay tuned for my next post, where I continue this discussion with Hand-Coding: What Went Wrong and How to Avoid Repeating History’s Mistakes


Link to original post

RickSherman November 4, 2009
Share this Article
Facebook Twitter Pinterest LinkedIn
Share
- Advertisement -

Follow us on Facebook

Latest News

cloud-centric companies using network relocation
Cloud-Centric Companies Discover Benefits & Pitfalls of Network Relocation
Cloud Computing
construction analytics
5 Benefits of Analytics to Manage Commercial Construction
Analytics
database compliance guide
Four Strategies For Effective Database Compliance
Data Management
Digital Security From Weaponized AI
Fortifying Enterprise Digital Security Against Hackers Weaponizing AI
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

What is Data Pipeline A detailed explaination
Big DataData LakeData ManagementData WarehousingExclusive

What is Data Pipeline? A Detailed Explanation

8 Min Read
etl for data-driven businesses
Big DataData WarehousingExclusive

Understanding ETL Tools as a Data-Centric Organization

8 Min Read
data lake vs data warehouse
Big DataData LakeData WarehousingExclusive

Differentiating Between Data Lakes and Data Warehouses

7 Min Read
moving to the cloud
Big DataCloud ComputingData WarehousingExclusive

How Will The Cloud Impact Data Warehousing Technologies?

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Artificial Intelligence Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?