Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Data Mining Theory vs. Practice
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Data Mining Theory vs. Practice
Data Mining

Data Mining Theory vs. Practice

DeanAbbott
DeanAbbott
3 Min Read
SHARE

In many fields, it is common to find a gap between theorists and practitioners. As stereotypes, theorists have a reputation for sniffing at anything which has not been optimized and proven to the nth degree, while practitioners show little interest in theory, as it “only ever works on paper”.

In many fields, it is common to find a gap between theorists and practitioners. As stereotypes, theorists have a reputation for sniffing at anything which has not been optimized and proven to the nth degree, while practitioners show little interest in theory, as it “only ever works on paper”.

I have been amazed at both extremes of this spectrum. Academic and standards journals seem to publish mostly articles which solve theoretical problems which will never arise in practice (but which permit solutions which are elegant or which can be optimized to some ridiculous level), or solutions which are trivial variations on previous work. The same goes for most masters and doctoral theses. On the other hand, I was shocked when software development colleagues (consultants: the last word in practice over theory) were unfamiliar with two’s complement arithmetic.

Data mining is certainly not immune to this problem. Not long ago, I came upon technical documentation for a linear regression which had been “fixed” by a logarithmic transformation of the dependent variable. (There is a correct way to fit coefficients in this circumstance, but that was not done in this case.) Even more astounding was the polynomial curve fit which was applied to “undo” the log transformation, to get back to the original units! Sadly, the practitioners in question did not even recognize the classic symptom of their error: residuals were much larger at the high end of their plots.

More Read

A Fool and Their Data are Soon Parted
Analyzing big data with Revolution R Enterprise
When sharing isn’t a good idea
Embracing Email Authentication a Must
Revealing Human Nature through Social Media Measurement

Data miners (statisticians, quantitative analysts, forecasters, etc.) come from a variety of fields, and enjoy diverse levels of formal training. Grounding in theory follows suit. The people we work for typically are capable of identifying only the most egregious technical errors in our work. This sets the stage for potential problems.

As a practitioner, I have found much that is useful in theory and suggest that it is a fountain which is worth returning to, from time to time. Reviewing new developments in our field, searching for useful techniques and guidance will benefit data miners, regardless of their seniority.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data analytics and truck accident claims
How Data Analytics Reduces Truck Accidents and Speeds Up Claims
Analytics Big Data Exclusive
predictive analytics for interior designers
Interior Designers Boost Profits with Predictive Analytics
Analytics Exclusive Predictive Analytics
big data and cybercrime
Stopping Lateral Movement in a Data-Heavy, Edge-First World
Big Data Exclusive
AI and data mining
What the Rise of AI Web Scrapers Means for Data Teams
Artificial Intelligence Big Data Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

According to CNNmoney.com, The acquisition of ILOG will…

1 Min Read

The Analytics of Terrorism

6 Min Read

At an event in its Hawthorne, NY research facility, Big Blue…

2 Min Read

Is there a voice quality testing software that can compare two audio files? Does it have advantages if compared with existing ones? – Yes, this software already exists! Yes it does have advantages!

2 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai chatbot
The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
Chatbots
ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?