Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    sales and data analytics
    How Data Analytics Improves Lead Management and Sales Results
    9 Min Read
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Target, Pregnancy, and Predictive Analytics, Part II
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Data Management > Best Practices > Target, Pregnancy, and Predictive Analytics, Part II
AnalyticsBest PracticesData MiningMarketingModelingStatistics

Target, Pregnancy, and Predictive Analytics, Part II

DeanAbbott
DeanAbbott
0 Min Read
SHARE

This is part II of my thoughts on the New York Times article “How Companies Learn Your Secrets”. 

This is part II of my thoughts on the New York Times article “How Companies Learn Your Secrets”. 

In the first post, I commented on the quote

“It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at Amazon.com. “Mathematicians are suddenly sexy.”

Comments on this can be seen in Part I here.

In this post, the next portion of the article I found fascinating can be summarized by the section that says

Habits aren’t destiny — they can be ignored, changed or replaced. But it’s also true that once the loop is established and a habit emerges, your brain stops fully participating in decision-making. So unless you deliberately fight a habit — unless you find new cues and rewards — the old pattern will unfold automatically.

Habits are what predictive models are all about. Or putting as a question, “is customer behavior predictable based on their past behavior?” The Frawley, Piatetsky-Shapiro, Mattheus definition of knowledge discovery in databases (KDD) as follows:

More Read

Top 10 Tips for Securely Managing BYOD in the Workplace
Analyst: “Every cloud needs SOA behind it:”
Interview: Karl Rexer – Rexer Analytics
HCIR: Better Than Magic!
How to Collect and Transform Data into Value for Your Business

Knowledge discovery is the nontrivial extraction of implicit, previously unknown,and potentially useful information from data. (PDF of the paper can be found here)

This quote has often been applied to data mining and predictive analytics, and rightfully so. We believe there are patterns hidden in the data and want to characterize those patterns with predictive modeols. Predictive models usually work best when individuals don’t even realize what they are doing so we can capture their behavior solely based on what they want to do rather than behavior influence by how they want to be perceived, which is exactly how the Target models were built.

So what does this have to do with the NYTimes quote? The “habits” that “unfold automatically” as described in the article was fascinating precisely because predictive models rely on habits; we wish to make the connection between past behavior and expected result as captured in the data that are consistent and repeatable (that is, habitual!). These expected results could be “is likely to respond to a mailing”, “is likely purchase a product online”, “is likely to commit fraud”, or in the case of the article, “is likely to be pregnant”. Duhigg (and presumably Pole describing it to Duhigg) characterizes this very well. The behavior Target measured was shoppers purchasing behavior when they were to give birth some weeks or months in the future, and nothing more. They had to apply broadly to thousands of “Guest IDs” for models to work effectively. 

The description of what Andy Pole did for target is an excellent summary of what predictive modelers can and should do. The approach included domain knowledge, understanding of what predictive models can do, and most of all a forensic mindset. I quote again from the article:

“Target has a baby-shower registry, and Pole started there,observing how shopping habits changed as a woman approached her due date, which women on the registry had willingly disclosed. He ran test after test, analyzing the data, and before long some useful patterns emerged. Lotions, for example. Lots of people buy lotion, but one of Pole’s colleaguesnoticed that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-big bags of cotton balls, in addition to hand sanitizers and washcloths, it signals they could be getting close to their delivery date.” (emphases mine)

To me, the key descriptive terms in the quote from the article are “observed“, “noticed” and “noted“. This means the models were not built as black boxes; the analysts asked “does this make sense?” and leveraged insights gained from the patterns found in the data to produce better predictive models. It undoubtedly was iterative; as they “noticed” patterns, they were prompted to consider other patterns they had not explicitly considered before (and maybe had not even occurred to them before). But it was these patterns that turned out to be the difference-makers in predicting pregnancy. 

So after all my preamble here, the key take-home messages from the article are: 
1) understand the data, 
2) understand why the models are focusing on particular input patterns, 
3) ask lots of questions (why does the model like these fields best? why not these other fields?)
4) be forensic (now that’s interesting or that’s odd…I wonder…),
5) be prepared to iterate, (how can we predict better for those customers we don’t characterize well)
6) be prepared to learn during the modeling process

We have to “notice” patterns in the data and connect them to behavior. This is one reason I like to build multiple models: different algorithms can find different kinds of patterns. Regression is a global predictor (one continuous equation for all data), whereas decision trees and kNN are local estimators. 

So we shouldn’t be surprised that we will be surprised, or put another way, we should expect to be surprised. The best models I’ve built contain surprises, and I’m glad they did!

 

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sales and data analytics
How Data Analytics Improves Lead Management and Sales Results
Analytics Big Data Exclusive
ai in marketing
How AI and Smart Platforms Improve Email Marketing
Artificial Intelligence Exclusive Marketing
AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

Google Experimenting with Social Search

5 Min Read
data analytics in strategic financial planning
Analytics

The Evolving Role of Analytics in Strategic Financial Management

6 Min Read

Here’s how decisions and rules relate (and how to manage them)

10 Min Read

The CIO Diaries – Bridging the Gap to LBOs

3 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?