By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    AI analytics
    AI-Based Analytics Are Changing the Future of Credit Cards
    6 Min Read
    data overload showing data analytics
    How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
    8 Min Read
    hire a marketing agency with a background in data analytics
    5 Reasons to Hire a Marketing Agency that Knows Data Analytics
    7 Min Read
    predictive analytics for amazon pricing
    Using Predictive Analytics to Get the Best Deals on Amazon
    8 Min Read
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Analyzing the Results of Analysis
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Data Management > Best Practices > Analyzing the Results of Analysis
AnalyticsBest PracticesCommentaryData QualityWeb Analytics

Analyzing the Results of Analysis

DeanAbbott
Last updated: 2011/03/29 at 4:43 PM
DeanAbbott
4 Min Read
SHARE

Sometimes, the output of analytical tools can be voluminous and complicated. Making sense of it sometimes requires, well, analysis. Following are two examples of applying our tools to their own output.

Model Deployment Verification

Sometimes, the output of analytical tools can be voluminous and complicated. Making sense of it sometimes requires, well, analysis. Following are two examples of applying our tools to their own output.

Model Deployment Verification

More Read

Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices

Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices

AI-Based Analytics Are Changing the Future of Credit Cards
Tips to Protect Office 365 Systems from Data Breaches
How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
5 Reasons to Hire a Marketing Agency that Knows Data Analytics

From time to time, I have deployed predictive models on a vertical application in the finance industry which is not exactly “user friendly”. I have virtually no access to the actual deployment and execution processes, and am largely limited to examination the production mode output, as implemented on the system in question.

As sometimes happens, the model output does not match my original specification. While the actual deployment is not my individual responsibility, it very much helps if I can indicate where the likely problem is. As these models are straightforward linear or generalized linear models (with perhaps a few input data transformations), I have found it useful to calculate the correlation between each of the input variables and the difference between the deployed model output and my own calculated model output. The logic is that input variables with a higher correlation with the deployment error are more likely to be calculated incorrectly. While this trick is not a cure-all, it quickly identifies in 80% or more of cases the culprit data elements.

Model Stability Over Time

A bedrock premise of all analytical work is that the future will resemble the past. After all, if the rules of the game keep changing, then there’s little point in learning them. Specifically in predictive modeling, this premise requires that the relationship between input and output variables must remain sufficiently stable for discovered models to continue to be useful in the future.

In a recent analysis, I discovered that models universally exhibited a substantial drop in test performance, when comparing out-of-time to (in-time) out-of-sample. The relationships between at least some of my candidate input variables and the target variable are presumably changing over time. In an effort to minimize this issue, I attempted to determine which variables were most susceptible. I calculated the correlation between each candidate predictor and the target, both for an early time-frame and for a later one.

My thinking was that variables whose correlation changed the most across time were the least stable and should be avoided. Note that I was looking for changes in correlation, and not whether correlations were strong or weak. Also, I regarded strengthening correlations just as suspect as weakening ones: The idea is for the model to perform consistently over time.

In the end, avoiding the use of variables which exhibited “correlation slide” did weaken model performance, but did ensure that performance did not deteriorate so drastically out-of-time.

Final Thought

It is interesting to see how useful analytical tools can be when applied to the analytical process itself. I note that solutions like the ones described here need not use fancy tools: Often simple calculations of means, standard deviation and correlations are sufficient.

DeanAbbott March 29, 2011
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
Best Practices Big Data Data Collection Data Management Privacy
data protection for SMEs
8 Crucial Tips to Help SMEs Guard Against Data Breaches
Data Management
How AI is Boosting the Customer Support Game
How AI is Boosting the Customer Support Game
Artificial Intelligence
AI analytics
AI-Based Analytics Are Changing the Future of Credit Cards
Analytics Artificial Intelligence Exclusive

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
Best PracticesBig DataData CollectionData ManagementPrivacy

Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices

7 Min Read
AI analytics
AnalyticsArtificial IntelligenceExclusive

AI-Based Analytics Are Changing the Future of Credit Cards

6 Min Read
office 365 data protection
Risk Management

Tips to Protect Office 365 Systems from Data Breaches

9 Min Read
data overload showing data analytics
Big Data

How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?

8 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?