We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
    chatgpt image jul 21, 2026, 04 34 30 pm
    4 Core Benefits of Predictive Maintenance after Vibration Analysis
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Data Darwinism: Market Driven Data Quality
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Data Darwinism: Market Driven Data Quality
Data QualityMarketing

Data Darwinism: Market Driven Data Quality

Jim MacLennan
Jim MacLennan
5 Min Read
Data Darwinism: Market Driven Data Quality
Photo by Counselling on Pixabay (https://pixabay.com/photos/business-shoes-bags-leather-goods-888146/)
SHARE

Just trying a little contrarian thought this week …

Have you ever noticed how much time and energy goes in to data validation?

I think it stems from visual forms development and the wide variety of clever data entry controls that are available – everyone wants to write an app that gets the oooo, cool! vote of approval. But how much of that energy spills over from value-added to feature creep?

Regex complexity at its finest …

When your IT peers are showing off their internally developed tools, or when internal departments put so much creativity into their departmental data collection apps, try stepping back for a moment and taking a look at the amount of development, documentation, training, and maintenance work that gets generated. These amazing, subtle, and visually compelling methods for gathering and validating data can become complex validation rules that try to guarantee that only pristine data is ever added to the list.

More Read

IT + Marketing: Innovation Through Collaboration in B2B Software Companies
IT + Marketing: Innovation Through Collaboration in B2B Software Companies
What If We Could Feel the Big Data Sugar Rush Faster?
Amazing Ways AI is Changing the Marketing Landscape
The Great Rift
Can Machine Learning Help Us Avoid Common Email Marketing Mistakes?

Is all of this really necessary? Is there real value-add to this approach? Often times the coding of validation rules is so complex that the code becomes fragile, and burdensome on future maintenance programmers. Another common problem – many specialized, departmental, and/or narrowly vertical applications have broad ranges of acceptable data – and the rules for permissible values need to be wildly flexible and adaptive.

But how about NOT validating the input? Why not let “market forces” take over?

I am talking about instances where people are trying to get data into a System That Makes Some Problem Visible – for example, a database of projects or technical resource requests that have to be prioritized, or financial data that has to successfully post into a centralized data collection / aggregation system.

It might be easier to just document the requirements for the data, and then let the best quality data survive …

For your Project / Resource Prioritization application, a project will not get added to the prioritization list until all the data is complete and correct. Even if it is complete, it helps to make the project description easy to understand, compelling, and business relevant – or else someone else will get the resources.

Your monthly data submission has to conform to these [data structure] rules. If it does not conform, it will be kicked out / flagged with errors. You are responsible for getting your data cleaned up and compliant with the specification, and your data submitted by [the deadline] – else your submission will be late.

Now, this does put pressure on us to document the data formats and requirements clearly – but this is probably faster and easier than creating a gallery of automated rule checkers to validate input. And, when the document is proven to be complete, correct, and sufficient (i.e. not too complex), it would make a pretty good spec for an automated data validation program.

Just a wacky idea – as system designers, we don’t have to control the world. Try making market forces work in your favor, just like content struggling for readership on the internet or new products looking for sales …

… may the cleanest data win!

Share This Article
Facebook Pinterest LinkedIn
Share
ByJim MacLennan
Follow:
Jim MacLennan is Senior Vice President and Chief Information Officer at IDEX Corporation, a Fortune 1000 manufacturer that sells highly engineered products in a variety of markets worldwide. MacLennan has responsibility for Corporate IT services for all IDEX business units, and also drives innovation through initiatives that leverage Information and Technology as growth drivers for the industrial manufacturing space. He regularly publishes his observations and insights on the intersection of business and technology - check out his work at www.cazh1.com.

Follow us on Facebook

Latest News

Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
Big Data Business Intelligence Exclusive Web Analytics
Flat editorial illustration: A SaaS-editorial scene showing product usage, support patterns, and renewal feedback informing a fle
Using Product Usage Analytics and Support Data to Build a B2B Product Roadmap
Business Intelligence Infographic
From Hiring To Payroll: Managing HR Data Across Business Systems -- AI-generated illustration
From Hiring To Payroll: Managing HR Data Across Business Systems
Big Data Business Intelligence Infographic
Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
Big Data Business Intelligence Exclusive

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Do you believe in Magic (Quadrants)?
Data Quality

Do you believe in Magic (Quadrants)?

14 Min Read
QR Codes and Social Media
Data QualitySocial Data

QR Codes and Social Media

5 Min Read
Big Social Data Can Unlock the Power of Engaged Viewers
AnalyticsBig DataBusiness IntelligenceDecision ManagementMarket ResearchMarketingModelingPredictive AnalyticsSentiment AnalyticsSocial DataSocial Media Analytics

Big Social Data Can Unlock the Power of Engaged Viewers

9 Min Read
How I Estimate (Social/Sentiment/Text Analytics) Market Size
AnalyticsMarketingSentiment Analytics

How I Estimate (Social/Sentiment/Text Analytics) Market Size

8 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

5 Great Tips for Using Data Analytics for Website UX
5 Great Tips for Using Data Analytics for Website UX
Big Data
How To Get An Award Winning Giveaway Bot
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?