Crosstabs, boxplots and scatterplots can be effectively used to find ‘important data’ amidst the growing tidal wave of information from mobile devices, websites, smart cards, credit cards, loyalty cards, and other sources. Finding what data to include in a modeling exercise with crosstabs, boxplots and scatterplots first requires a clearly defined business problem.
Crosstabs, boxplots and scatterplots can be effectively used to find ‘important data’ amidst the growing tidal wave of information from mobile devices, websites, smart cards, credit cards, loyalty cards, and other sources. Finding what data to include in a modeling exercise with crosstabs, boxplots and scatterplots first requires a clearly defined business problem. That’s because the importance of a data variable may be a direct function of the modeling question that is to be answered: which customers are most likely to leave, what do my most profitable customers look like, in what areas of the country are our sales highest. Only after a clear modeling question is defined can a target variable be created in the datafile: whether this be a decile variable ‘PROFITABLE?’, boolean data point ‘ATTRITION?’ or a ratio variable named ‘PENETRATION RATE?’.
Once a target variable is defined, crosstabs, boxplots and scatterplots can help identify independent variables that should be included in a modeling exercise: logistic regression, inductive decision tree (IDT) or otherwise. Every independent variable must be evaluated on its own to see what significant impact, if any, it has on the target variable’s results.
For example, let’s assume that of the 199 independent variables in a datafile, 100 are numerical (N), 99 are categorical (C), and that a target variable ‘ATTRITION?’ has been created. A crosstab and a CHI-SQUARE test may be used to evaluate each of the 99 categorical variables against ATTRITION? according to how much it increases the likelihood of a customer attriting. T-TESTS, ANOVAS and MANOVAS might be used to analyze each of the 100 independent numerical variables for their impact on the likelihood of a customer attrition occurring.
Let’s assume that, after this data finding exercise, each of the following variables were found to have a significant impact on the likelihood of a customer attriting:
Age (N)
Gender (C)
Income (N)
Education (C)
Marital Status (C)
Average $ Purchase (N)
From a sea of 200 data variables, 6 were deemed important enough to include in a logistic regression, decision tree, neural network, or other datamining algorithm. Crosstabs, scatterplots and boxplots should also be applied to the smaller dataset in order to learn whether any of these 6 variables are highly correlated, as two highly-related variables should not be included in a model.
Once this is accomplished, the creative excercise of modeling may begin to answer the business question that was defined.
