By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Data Darwinism: Market Driven Data Quality
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Data Darwinism: Market Driven Data Quality
Data QualityMarketing

Data Darwinism: Market Driven Data Quality

Jim MacLennan
Last updated: 2011/02/20 at 3:44 PM
Jim MacLennan
5 Min Read
SHARE

Just trying a little contrarian thought this week …

Have you ever noticed how much time and energy goes in to data validation?

I think it stems from visual forms development and the wide variety of clever data entry controls that are available – everyone wants to write an app that gets the oooo, cool! vote of approval. But how much of that energy spills over from value-added to feature creep?

More Read

ai in ppc advertising

5 Proven Tips for Utilizing AI with PPC Advertising in 2023

Use this Strategic Approach to Maximize Your Data’s Value
14 Brands Using Mobile Apps Instead of Ads to Build Customer Loyalty
Niche Data Tactics to Take Your Business to the Next Level
Amazing Ways AI is Changing the Marketing Landscape

Just trying a little contrarian thought this week …

Have you ever noticed how much time and energy goes in to data validation?

I think it stems from visual forms development and the wide variety of clever data entry controls that are available – everyone wants to write an app that gets the oooo, cool! vote of approval. But how much of that energy spills over from value-added to feature creep?

Regex complexity at its finest …

When your IT peers are showing off their internally developed tools, or when internal departments put so much creativity into their departmental data collection apps, try stepping back for a moment and taking a look at the amount of development, documentation, training, and maintenance work that gets generated. These amazing, subtle, and visually compelling methods for gathering and validating data can become complex validation rules that try to guarantee that only pristine data is ever added to the list.

Is all of this really necessary? Is there real value-add to this approach? Often times the coding of validation rules is so complex that the code becomes fragile, and burdensome on future maintenance programmers. Another common problem – many specialized, departmental, and/or narrowly vertical applications have broad ranges of acceptable data – and the rules for permissible values need to be wildly flexible and adaptive.

But how about NOT validating the input? Why not let “market forces” take over?

I am talking about instances where people are trying to get data into a System That Makes Some Problem Visible – for example, a database of projects or technical resource requests that have to be prioritized, or financial data that has to successfully post into a centralized data collection / aggregation system.

It might be easier to just document the requirements for the data, and then let the best quality data survive …

For your Project / Resource Prioritization application, a project will not get added to the prioritization list until all the data is complete and correct. Even if it is complete, it helps to make the project description easy to understand, compelling, and business relevant – or else someone else will get the resources.

Your monthly data submission has to conform to these [data structure] rules. If it does not conform, it will be kicked out / flagged with errors. You are responsible for getting your data cleaned up and compliant with the specification, and your data submitted by [the deadline] – else your submission will be late.

Now, this does put pressure on us to document the data formats and requirements clearly – but this is probably faster and easier than creating a gallery of automated rule checkers to validate input. And, when the document is proven to be complete, correct, and sufficient (i.e. not too complex), it would make a pretty good spec for an automated data validation program.

Just a wacky idea – as system designers, we don’t have to control the world. Try making market forces work in your favor, just like content struggling for readership on the internet or new products looking for sales …

… may the cleanest data win!

Jim MacLennan February 20, 2011
Share this Article
Facebook Twitter Pinterest LinkedIn
Share
By Jim MacLennan
Follow:
Jim MacLennan is Senior Vice President and Chief Information Officer at IDEX Corporation, a Fortune 1000 manufacturer that sells highly engineered products in a variety of markets worldwide. MacLennan has responsibility for Corporate IT services for all IDEX business units, and also drives innovation through initiatives that leverage Information and Technology as growth drivers for the industrial manufacturing space. He regularly publishes his observations and insights on the intersection of business and technology - check out his work at www.cazh1.com.

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

ai in ppc advertising
Artificial Intelligence

5 Proven Tips for Utilizing AI with PPC Advertising in 2023

10 Min Read
analyzing big data for its quality and value
Big Data

Use this Strategic Approach to Maximize Your Data’s Value

6 Min Read
businesses using mobiles app for success
Big Data

14 Brands Using Mobile Apps Instead of Ads to Build Customer Loyalty

19 Min Read
niche data tactics for business success
Big Data

Niche Data Tactics to Take Your Business to the Next Level

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?