Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Data Integration: Hand-coding Using ETL Tools Part 2
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Uncategorized > Data Integration: Hand-coding Using ETL Tools Part 2
Uncategorized

Data Integration: Hand-coding Using ETL Tools Part 2

RickSherman
RickSherman
5 Min Read
SHARE

Hand-stop This is a continuation of an earlier post that discussed the problems of hand-coding using ETL tools.

What Went Wrong?

There are two aspects of effectively leveraging an ETL tool. First is learning the tool’s mechanics, for example, by taking the tool vendors’ training either in a class or through their on-line tutorials. Most IT people have no problem learning a tool’s syntax. Since they most likely already know SQL, they learn the tool very quickly.

But the second aspect actually involves understanding ETL processes. This includes knowing the data-integration processes needed to gather, conform, cleanse and transform; understanding not only what is dimensional modeling but why and how do you deploy it; being able to implement slowly changing dimensions (SCD) and change data capture (CDC); understanding the data demands of business intelligence; and being able to implement error handling and conditional processing.

More Read

Early Stage IR / NLP Investment Opportunity
Banks and the New Amazon Bank
Encrypting Your Life: Tools and Tips
Customer-Focused Marketing: Automation Is the Easy Part
@socialmedia2day Webinar: Inside Google+ w/ @MaggieFox

Without understanding the why of ETL processing, IT developers either quickly become disillusioned with ETL tools or simply under utilize them. Typically these ETL implementations merely result in the ETL tools executing SQL scripts or stored procedures, for example …



Hand-stop This is a continuation of an earlier post that discussed the problems of hand-coding using ETL tools.

What Went Wrong?

There are two aspects of effectively leveraging an ETL tool. First is learning the tool’s mechanics, for example, by taking the tool vendors’ training either in a class or through their on-line tutorials. Most IT people have no problem learning a tool’s syntax. Since they most likely already know SQL, they learn the tool very quickly.

But the second aspect actually involves understanding ETL processes. This includes knowing the data-integration processes needed to gather, conform, cleanse and transform; understanding not only what is dimensional modeling but why and how do you deploy it; being able to implement slowly changing dimensions (SCD) and change data capture (CDC); understanding the data demands of business intelligence; and being able to implement error handling and conditional processing.

Without understanding the why of ETL processing, IT developers either quickly become disillusioned with ETL tools or simply under utilize them. Typically these ETL implementations merely result in the ETL tools executing SQL scripts or stored procedures, for example, hand-coding.

These hand-coded processes within ETL tools are big trouble-makers. First, the tools have built-in transforms such as SCD and CDC which, if you don’t use, make you re-invent the wheel (code something you already bought). In doing so, you’re likely doing something inefficient at best and outright wrong at worst.

Second, ETL tools are built to be more efficient at extracting, transforming and loading data than SQL coders.

Third, the IT staff is not likely to code extensive error handling or audit routines that are pre-built in the ETL tools. This lessens productivity and responsiveness to issues in data quality.

Fourth, hand-coded processes are often not documented or, if they are initially, they’re not likely to be maintained.

Finally, each hand-coded operation is a custom job that each new developer has to learn, versus being able to bring in a developer who knows an ETL tool.

How to Avoid Repeating History’s Mistakes

You don’t know what you don’t know. It’s not that the IT staff wants to use these ETL tools either incorrectly or inappropriately, but they don’t know any better.

I’ll keep preaching that data-integration processes should be developed using ETL tools rather than hand coding. But what I have learned along the way is I also need to advocate that anyone using these tools learn not just about the tool but more importantly about ETL processing.

FYI: A good starting place is my articles on ETL. Check out my corporate library pointing to my articles, posts, webinars, podcasts and white papers.


Link to original post

TAGGED:etl tools
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive
data analytics and truck accident claims
How Data Analytics Reduces Truck Accidents and Speeds Up Claims
Analytics Big Data Exclusive
predictive analytics for interior designers
Interior Designers Boost Profits with Predictive Analytics
Analytics Exclusive Predictive Analytics

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

No Shortcuts: Focus on DW/BI Architecture and Processes, then Think about the Tools

4 Min Read

Tools and those who enable their misuse

7 Min Read

ETL tools: Don’t Forget About the Little Dogs

6 Min Read

#25: Here’s a thought…

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?