# Keeping Your Big Data Analysis Clean

‘Outlier’ is a term that comes from statistics and data analytics. Math.com defines an outlier as “a value that lies outside (is much smaller or larger than) most of the other values in a set of data,” and it gives a sample of values for an example. If you have the values 25, 29, 3, 32, 85, 33, 27, and 28, both 3 and 85 are your outliers.”

‘Outlier’ is a term that comes from statistics and data analytics. Math.com defines an outlier as “a value that lies outside (is much smaller or larger than) most of the other values in a set of data,” and it gives a sample of values for an example. If you have the values 25, 29, 3, 32, 85, 33, 27, and 28, both 3 and 85 are your outliers.”

If you’re looking to become an outlier, and/or if you’re using a big data platform to optimize your entire business model—looking for outliers to either weed out or leverage—then it’s important to understand where outliers come from and how instructive or beneficial they are to your particular data set. Above all else, you must learn to recognize whether any outliers that crop up in your data analysis are the result of flaws in your analytics model, or if they’re anomalies particular to your specific business, and whether they’re something to be eliminated or enhanced. That level of understanding begins and ends with keeping your big data analysis clean. Here are a few tips for that.

1. Investigate and identify the cause — Not all outliers are the result of errors. They may be exactly what you’re looking for. However, sometimes outliers come from a transcription error or malfunctioning equipment that is reporting inaccurate values. Extreme outliers like these can negatively impact the accuracy of your analysis. So, you’ll want to either remove these values from the data set or fix the flaws causing them.

1. Use data visualization tools — Data visualization tools make looking at trends and patterns in a large data set much easier than just looking at the numbers. Seeing anomalies is the first step to understanding them.

1. Know the factors that may skew your data — In a bar full of average people plus Bill Gates, a measurement on the average income in the room would be skewed by the presence of Gates. French census data taken the year Napoleon Bonaparte was born would show nothing out of the ordinary, and yet, how much did he impact European census data throughout the nineteenth century? These examples show how outliers can heavily influence average values.

1. Be agnostic about your outliers — In and of themselves, outliers are neither good nor bad. They are simply extreme values that may or may not be expected. Most of all, outliers are instructive. They represent risks, opportunities, mistakes, anomalies, or something else. Their usefulness is a product of context and how that relates to a company’s goals.