Usually data miners don’t cheat. The reason is simple: you cannot cheat with the future. In reality, it’s a bit more complicated. A data miner may be cheating without knowing it. Here are a few examples:
First, one may cheat by learning the training set by heart. If you cheat (in any way) on your training set, it will certainly be visible on the test set (overfitting).
Another way of cheating is to use predictor variables that you don’t have the right to use. For example, you don’t have the right to use the value of the euro of tomorrow to predict the value of the dollar of tomorrow). Straightforward, but this may happen in more subtle ways, believe me.
The next one is much more insidious. You use past data to predict the evolution of some given stocks. You train and test our model on past years (backtesting). For that, a set of stocks are choosen (as of end 2010), that were also present in 2001. This way, backtest can be performed on 10 years. Results are good over the last 10 years, so the strategy can be launched.
But you are cheating, in a subtle way. You are not overfitting and neither are you using forbidden predictor variables. So, what’s the problem? You are using non-allowed information. While backtesting, will should not know in 2001 which companies will still be there in the end of 2010. By doing so, only successful companies are selected and thus you cheat on your predicted return. Subtle isn’t it?
