We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
    From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
    12 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Tweety Bird and Aha! Moments
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Inside Companies > Tweety Bird and Aha! Moments
Inside Companies

Tweety Bird and Aha! Moments

MIKE20
MIKE20
4 Min Read
Tweety Bird and Aha! Moments
Photo by Daniel_Nebreda on Pixabay (https://pixabay.com/photos/women-one-wall-minimalism-bank-3118387/)
SHARE

About three months ago, I started a data management and ETL project for a pretty big bank. Time was of the essence and the bank brought me in because I can get results. In this post, I explain why an overemphasis on results can be a really bad thing–and why all matching isn’t created equal.When my client advised me of the number of disparate extracts of financial data I would receive, I quickly looked for potential commonalities. As most would do on this type of project, I started with the obvious: GL account. While not unique in most systems I have seen, they are often at least part of a multi-field index. In other words, I can almost always sum debits and credits by GL account 1000, for example, especially when I bring in a field like company.

Dealing with Suboptimal Data and Other Limitations

Unfortunately, a few of the extracts only contained account descriptions. When I explained this limitation to my boss, we had the following exchange:

Boss: Well, why not join on description instead of account number?

More Read

Data Mining Improved Company’s Revenue By 187%
Data Mining Improved Company’s Revenue By 187%
What Aspiring Data Scientists Are Looking For in Hiring Companies
Why is there so little innovation in cloud hardware?
On Business Intelligence and Real-Time Intelligence
#10: Here’s a thought…

Me: It’s possible, but I never recommend it. In many systems, these descriptions are not standardized. One missing character, misplaced space, or mutant letter will result in missing data downstream and other problems that are worth trying to address from the beginning.

Boss: We have no other choice. IT won’t change the extract from XYZ financial system. Gotta make do….

Me (in Tweety Bird voice): OK…but I got a baaaaaad feeling about this.

Fast forward to the end of the project. Our reports were off–way off. What’s more, this wasn’t a fundamental design issue. So far off that I had to rejigger a bunch of previously working queries and validation routines. This took a few days and caused other problems. We had to break things that previously worked.

Sometimes we don’t have a choice in life and in golf. We hit the ball into the woods, can’t find it, have to take a penalty stroke, and walk away with a snowman on a par 3. It happens.

Although my client wasn’t happy that we had so many problems, he fully understood what I told him on day one: “Try not to join on descriptions.” Of course, the bell didn’t ring quite as loud when I initially said that. After he saw the results first-hand, that bell was pretty loud–and wouldn’t stop.

Understand that there are limitations with certain types of data matching, as many on this blog has pointed. Some types are much better than others. Some are too restrictive; others are far too liberal. Think about it.

As one of my friends put so nicely, you don’t go to pick up your kid at nursery school and just grab any eight-year old brunette girl. You probably want you own kid.

Data’s the same way.

What say you?

Read more at MIKE2.0: The Open Source Standard for Information Management

TAGGED:data quality
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article connects downstream report failures, data-quality gaps, and conflicting metric definitio
Signs Your Data Architecture Needs an Outside Review
Big Data Exclusive
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Big Data Exclusive
Flat editorial illustration: The article contrasts tools for gathering fresh interviews with tools for analyzing existing qualita
10 AI Market Research Tools That Turn Qualitative Data into Insights
Artificial Intelligence Big Data Exclusive
Flat editorial illustration: The article compares proxy MCP servers by what an AI agent can do through them: manage proxy account
Proxy MCP Servers: 7 Options and What Each Lets an AI Agent Do
Artificial Intelligence Big Data Data Collection

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Flat editorial illustration: A technology-editorial comparison of disconnected origination, servicing, and collections records dr
Big DataExclusiveFintech

The Data Reconciliation Problem Inside Disconnected Lending Systems

19 Min Read
Analytics: Not About Saving Time
Analytics

Analytics: Not About Saving Time

7 Min Read
#7: Here's a thought...
Business IntelligenceCRMData Visualization

#7: Here’s a thought…

7 Min Read
Stuck in First Gear
Business IntelligenceCRMData MiningData Visualization

Stuck in First Gear

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
How To Get An Award Winning Giveaway Bot
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?