By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Statisticians Push Back Against the “End of Theory” Problem
Share
Notification Show More
Latest News
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Statistics > Statisticians Push Back Against the “End of Theory” Problem
Statistics

Statisticians Push Back Against the “End of Theory” Problem

tkorte
Last updated: 2013/08/28 at 8:00 AM
tkorte
3 Min Read
Image
SHARE

For years, some commentators have worried that increasing volumes of data coupled with better and better automated prediction methods would lead to an “end of theory.” What they mean is that the sorts of insights traditional statisticians like to be able to infer from their models of the world (those observations that can be generalized and applied to other problems) are often absent from machine learning algorithms that automatically select hundreds or thousands of parameters.

For years, some commentators have worried that increasing volumes of data coupled with better and better automated prediction methods would lead to an “end of theory.” What they mean is that the sorts of insights traditional statisticians like to be able to infer from their models of the world (those observations that can be generalized and applied to other problems) are often absent from machine learning algorithms that automatically select hundreds or thousands of parameters. The machine learning methods often work extraordinary well for prediction, but they only give answers—they do not teach lessons.

ImageDr. Ryan Tibshirani, an Assistant Professor of statistics at Carnegie Mellon University, is trying to fix that. Tibshirani and his colleagues (including his father, famed statistical methodologist Dr. Robert Tibshirani) have developed a new method that hopes to satisfy both the prediction and inference sides of statistics, offering traditional statisticians insights while preserving the adaptability and predictive power of modern machine learning methods.

The machine learning technique they tackled is known as the lasso method, a widely used automated method that ensures models do not get too elaborate. The greatest enemy of predictive analytics (particularly in the “big data” arena) is overfitting, which occurs when a model adheres too closely to a given dataset and becomes less accurate when it is applied to new data; the lasso method helps keep models simpler and more extensible. The problem with the lasso method was that standard significance tests—which help statisticians determine whether a variable is really important or can be thrown out of the model—did not work on it, meaning that it was unable to produce some of the inferential contributions statisticians often demand.

Tibshirani and his colleagues developed a special significance test just for the lasso method (the technical details of which can be found here), and have pointed the way to future research into adding inferential capabilities to other predictive modeling techniques. Although this is only the first step, the promise of more insightful algorithmic methods is exciting. In complex environments such as biological and urban systems, the profusion of variables that might be contributing to a particular effect is enormous, and the value of “big data” prediction paired with generalizable inference may be great as well.

TAGGED: end of theory
tkorte August 28, 2013
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data
ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?