Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
    pexels pavel danilyuk 8112119
    Data Analytics Is Revolutionizing Medical Credentialing
    8 Min Read
    data and seo
    Maximize SEO Success with Powerful Data Analytics Insights
    8 Min Read
    data analytics for trademark registration
    Optimizing Trademark Registration with Data Analytics
    6 Min Read
    data analytics for finding zip codes
    Unlocking Zip Code Insights with Data Analytics
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Statisticians Push Back Against the “End of Theory” Problem
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Statistics > Statisticians Push Back Against the “End of Theory” Problem
Statistics

Statisticians Push Back Against the “End of Theory” Problem

tkorte
tkorte
3 Min Read
Image
SHARE

For years, some commentators have worried that increasing volumes of data coupled with better and better automated prediction methods would lead to an “end of theory.” What they mean is that the sorts of insights traditional statisticians like to be able to infer from their models of the world (those observations that can be generalized and applied to other problems) are often absent from machine learning algorithms that automatically select hundreds or thousands of parameters.

For years, some commentators have worried that increasing volumes of data coupled with better and better automated prediction methods would lead to an “end of theory.” What they mean is that the sorts of insights traditional statisticians like to be able to infer from their models of the world (those observations that can be generalized and applied to other problems) are often absent from machine learning algorithms that automatically select hundreds or thousands of parameters. The machine learning methods often work extraordinary well for prediction, but they only give answers—they do not teach lessons.

ImageDr. Ryan Tibshirani, an Assistant Professor of statistics at Carnegie Mellon University, is trying to fix that. Tibshirani and his colleagues (including his father, famed statistical methodologist Dr. Robert Tibshirani) have developed a new method that hopes to satisfy both the prediction and inference sides of statistics, offering traditional statisticians insights while preserving the adaptability and predictive power of modern machine learning methods.

The machine learning technique they tackled is known as the lasso method, a widely used automated method that ensures models do not get too elaborate. The greatest enemy of predictive analytics (particularly in the “big data” arena) is overfitting, which occurs when a model adheres too closely to a given dataset and becomes less accurate when it is applied to new data; the lasso method helps keep models simpler and more extensible. The problem with the lasso method was that standard significance tests—which help statisticians determine whether a variable is really important or can be thrown out of the model—did not work on it, meaning that it was unable to produce some of the inferential contributions statisticians often demand.

More Read

Conducting Remote User Research: 10 Key Concepts
Executives Don’t Like Analytics: Why Business Isn’t Data-Driven
Analytics, Graph Search, APIs: Is Facebook Struggling with Big Data?
Your Statistical Result is Below .05: Now What?
The Data Analytics of Gas vs. Charcoal Grills Revisited

Tibshirani and his colleagues developed a special significance test just for the lasso method (the technical details of which can be found here), and have pointed the way to future research into adding inferential capabilities to other predictive modeling techniques. Although this is only the first step, the promise of more insightful algorithmic methods is exciting. In complex environments such as biological and urban systems, the profusion of variables that might be contributing to a particular effect is enormous, and the value of “big data” prediction paired with generalizable inference may be great as well.

TAGGED:end of theory
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

accountant using ai
AI Improves Integrity in Corporate Accounting
Exclusive
ai and law enforcement
Forensic AI Technology is Doing Wonders for Law Enforcement
Artificial Intelligence Exclusive
langgraph and genai
LangGraph Orchestrator Agents: Streamlining AI Workflow Automation
Artificial Intelligence Exclusive
ai fitness app
Will AI Replace Personal Trainers? A Data-Driven Look at the Future of Fitness Careers
Artificial Intelligence Big Data Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?