By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Data Integration: Hand-coding Using ETL Tools Part 2
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Uncategorized > Data Integration: Hand-coding Using ETL Tools Part 2
Uncategorized

Data Integration: Hand-coding Using ETL Tools Part 2

RickSherman
Last updated: 2009/11/06 at 1:26 PM
RickSherman
5 Min Read
SHARE

Hand-stop This is a continuation of an earlier post that discussed the problems of hand-coding using ETL tools.

What Went Wrong?

There are two aspects of effectively leveraging an ETL tool. First is learning the tool’s mechanics, for example, by taking the tool vendors’ training either in a class or through their on-line tutorials. Most IT people have no problem learning a tool’s syntax. Since they most likely already know SQL, they learn the tool very quickly.

But the second aspect actually involves understanding ETL processes. This includes knowing the data-integration processes needed to gather, conform, cleanse and transform; understanding not only what is dimensional modeling but why and how do you deploy it; being able to implement slowly changing dimensions (SCD) and change data capture (CDC); understanding the data demands of business intelligence; and being able to implement error handling and conditional processing.

More Read

etl for data-driven businesses

Understanding ETL Tools as a Data-Centric Organization

Tools and those who enable their misuse
No Shortcuts: Focus on DW/BI Architecture and Processes, then Think about the Tools
ETL tools: Don’t Forget About the Little Dogs
#25: Here’s a thought…

Without understanding the why of ETL processing, IT developers either quickly become disillusioned with ETL tools or simply under utilize them. Typically these ETL implementations merely result in the ETL tools executing SQL scripts or stored procedures, for example …



Hand-stop This is a continuation of an earlier post that discussed the problems of hand-coding using ETL tools.

What Went Wrong?

There are two aspects of effectively leveraging an ETL tool. First is learning the tool’s mechanics, for example, by taking the tool vendors’ training either in a class or through their on-line tutorials. Most IT people have no problem learning a tool’s syntax. Since they most likely already know SQL, they learn the tool very quickly.

But the second aspect actually involves understanding ETL processes. This includes knowing the data-integration processes needed to gather, conform, cleanse and transform; understanding not only what is dimensional modeling but why and how do you deploy it; being able to implement slowly changing dimensions (SCD) and change data capture (CDC); understanding the data demands of business intelligence; and being able to implement error handling and conditional processing.

Without understanding the why of ETL processing, IT developers either quickly become disillusioned with ETL tools or simply under utilize them. Typically these ETL implementations merely result in the ETL tools executing SQL scripts or stored procedures, for example, hand-coding.

These hand-coded processes within ETL tools are big trouble-makers. First, the tools have built-in transforms such as SCD and CDC which, if you don’t use, make you re-invent the wheel (code something you already bought). In doing so, you’re likely doing something inefficient at best and outright wrong at worst.

Second, ETL tools are built to be more efficient at extracting, transforming and loading data than SQL coders.

Third, the IT staff is not likely to code extensive error handling or audit routines that are pre-built in the ETL tools. This lessens productivity and responsiveness to issues in data quality.

Fourth, hand-coded processes are often not documented or, if they are initially, they’re not likely to be maintained.

Finally, each hand-coded operation is a custom job that each new developer has to learn, versus being able to bring in a developer who knows an ETL tool.

How to Avoid Repeating History’s Mistakes

You don’t know what you don’t know. It’s not that the IT staff wants to use these ETL tools either incorrectly or inappropriately, but they don’t know any better.

I’ll keep preaching that data-integration processes should be developed using ETL tools rather than hand coding. But what I have learned along the way is I also need to advocate that anyone using these tools learn not just about the tool but more importantly about ETL processing.

FYI: A good starting place is my articles on ETL. Check out my corporate library pointing to my articles, posts, webinars, podcasts and white papers.


Link to original post

TAGGED: etl tools
RickSherman November 6, 2009
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

etl for data-driven businesses
Big Data

Understanding ETL Tools as a Data-Centric Organization

8 Min Read

Tools and those who enable their misuse

7 Min Read

No Shortcuts: Focus on DW/BI Architecture and Processes, then Think about the Tools

4 Min Read

ETL tools: Don’t Forget About the Little Dogs

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?