Sign up | Login with →

Posts by Dean Abbott Subscribe

Predictive Modeling Skills: Expect to be Surprised

Excerpted from Chapter 1 of my book Applied Predictive Analytics, Wiley 2014Conventional wisdom says that predictive modelers need to have an academic background in statistics, mathematics, computer science, or engineering. A degree in one of these...

Posted December 7, 2015    

Similarities and Differences Between Predictive Analytics and Business Intelligence

I’ve been reminded recently of the overlap between business intelligence and predictive analytics. Of course any reader of this blog (or at least the title of the blog) knows I live in the world of data mining (DM) and predictive analytics (PA), not...

Posted August 12, 2014    

Why Overfitting is More Dangerous than Just Poor Accuracy [PART 2]

In Part I, I explained one problem with overfitting the data: estimates of the target variable in regions without any training data can be unstable, whether those regions require the model to interpolate or extrapolate. Accuracy is a problem,...

Posted June 9, 2014    

Why Overfitting is More Dangerous than Just Poor Accuracy [PART 1]

Arguably, the most important safeguard in building predictive models is complexity regularization to avoid overfitting the data. When models are overfit, their accuracy is lower on new data that wasn’t seen during training, and therefore when these...

Posted June 6, 2014    

A Good Business Objective Beats a Good Algorithm

Predictive Modeling competitions, once the arena for a few data mining conferences, has now become big business. Kaggle ( is perhaps the most well-known forum for modeling competitions, using a crowd-sourcing mentality: if more people try...

Posted November 26, 2013    

The NSA, Link Analysis and Fraud Detection

The recent leaks about the NSA’s use data mining and predictive analytics has certainly raised awareness of our field and has resulted in hours of discussions with family, relatives, friends and reporters about what predictive analytics can (and can...

Posted July 25, 2013    

Big Data Is Not Enough

Big data is the big buzz word in the world of analytics today. According to google trends, shown in the figure, searches for "big data" have been growing exponentially since 2010 though perhaps is beginning to level off. Or take a look on

Posted June 18, 2013    

Do Predictive Modelers Need to Know Math?

Predictive analytics is just a bunch of math, isn’t it? After all, algorithms in the form of matrix algebra, summations, integrals, multiplies and adds are the core of what predictive modeling algorithms do. Even rule-based approaches need math to...

Posted April 2, 2013    

Takeaways From Your Next Predictive Analytics Conference

Why should one go to a predictive analytics conference? What should one take home from a conference like Predictive Analytics World (PAW)? There are many reasons conferences are valuable, including interacting with thought leaders and practitioners...

Posted February 15, 2013    

Using Geographic Data

Most organizations collect and maintain some type of geographic data, yet many ignore this data during analysis. Any business has some record of customer addresses, for instance, but this data is usually formatted in an awkward, non-numeric form....

Posted February 11, 2013    

Data Mining and Analysis Aren't Always the Answer

Data mining is an important tool whose benefits have been demonstrated in diverse fields, among business, government and non-profit organizations. Its application areas continue to grow, especially given the ever-shrinking cost of gathering and...

Posted February 4, 2013    

Three Ways to Get Your Predictive Models Deployed

We all know that given reasonable data, a good predictive modeler can build a model that works well and helps make makes better decisions than what is currently used in your organization (at least in our own minds). Newer data, sophisticated...

Posted January 19, 2013