By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Data Integration Processes: It’s Not the Tool, It’s How You Use It
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Data Integration Processes: It’s Not the Tool, It’s How You Use It
Collaborative DataData QualityData Visualization

Data Integration Processes: It’s Not the Tool, It’s How You Use It

RickSherman
Last updated: 2011/09/23 at 11:36 AM
RickSherman
4 Min Read
SHARE
- Advertisement -

Tools In my discussions with clients, prospects, students and networking with folks at seminars I am always asked about my opinion or recommendations on data integration and ETL (extract, transform and load) products.

Tools In my discussions with clients, prospects, students and networking with folks at seminars I am always asked about my opinion or recommendations on data integration and ETL (extract, transform and load) products. People always like to talk products and much of industry literature is centered on tools.

I’m happy to discuss products, but every once in a while someone asks me a more insightful question,  which is what happened this week. That person asked what the main shortcomings or stumbling blocks are that companies encounter when implementing data integration.

Great question. I discuss this when I am working with clients and teaching courses, but hardly anyone asks me that and directs the discussion in that direction.

More Read

data integration guide

How AI and ML Can Transform Data Integration

5 Reasons Technical Support is Essential in the Big Data Age
Why You Need A Methodology For Your Big Data Research
5 Ways to Make Big Data Investment Work For Your Organization
Two Ways GPU Databases Are Transforming the Retail Industry

My answer is simple: it’s not the tool, but how you use it that determines success. Although you do have to know the mechanics of the tool that is not the critical success factor. What really matters is the mechanics of data integration.

Many people don’t understand data integration processes and the frameworks products provide to implement those processes. And it’s not just data integration newbies that have this problem; it’s also experienced veterans.

Most data integration architects, designers and developers started ETL by writing SQL scripts or manually coding using something like Java with JDBC. Then they try to replicate what they did in the manual code into the data integration processes. This is probably the worst way to use a data integration product!  You likely get little benefit from the framework, processing is not optimized (maybe even terrible) and worse, the developer gets frustrated because he feels he could have coded it faster.

Welcome to the world of frustrated data integration processes, where people either assume these products are not useful or that the particular product they used must not be very good.

Almost all data integration products provide data imports/exports; data and workflows; data transformations; error handling; monitoring; performance tuning; and many processes that have evolved as best practices such as slowly changing dimensions (SCD), change data capture (SCD),  and hierarchy management.  All of these pre-built capabilities mean that data integration development does not have to reinvent the wheel, but can leverage industry best practices to develop world-class integration. But instead many data integration developers are spending their time creating the equivalent of manually coded import, extract and transforms without ever having time to get the best practices that would best serve their business.

Any successful, productive, robust data integration effort needs people who understand the necessary processes and can implement best practices.  Getting the tool and having the people who know how to use the tool is only the beginning.  You will get nowhere fast until you make sure you have people who understand data integration processes.


   

TAGGED: data integration
RickSherman September 23, 2011
Share this Article
Facebook Twitter Pinterest LinkedIn
Share
- Advertisement -

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data integration guide
Artificial Intelligence

How AI and ML Can Transform Data Integration

6 Min Read
technical support in the age of big data
Big Data

5 Reasons Technical Support is Essential in the Big Data Age

8 Min Read
Big Data Research
Best PracticesBig Data

Why You Need A Methodology For Your Big Data Research

6 Min Read
Big Data Investment
AnalyticsBig DataExclusive

5 Ways to Make Big Data Investment Work For Your Organization

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?