Data Mining Book Review: Dance with Chance

If you ever worked on time series prediction (forecasting), you should read Dance with Chance. It is written by a statistician, a psychologist and a decision scientist (Makriddakis, Hogarth and Gaba). As it is the case in The Numerati or Super Crunchers, authors explain complex notions to a non-expert audience. I find the book really interesting and provocative.

The main concept of Dance with Chance is the “illusion of control”. It is when you think you control a future event or situation, that is in fact mainly due to chance. This is the opposite of fatalism (when you think you have no control, although you have). The book teaches how to avoid being fooled by this illusion of control. This is a very interesting reading for any data miner, particularly involved with forecasting. The books contains dozens of examples of the limitation of forecasting techniques. For example, it explains the issues of forecasting the stock market and when predictions are due to chance. Authors use a brilliant mix of statistics and psychology to prove their point.

Of course, it is difficult for someone in the field to completely agree with the authors. For example, they often state that no one can predict the future in advance. Formulated this way, one may agree. However, data mining and machine learning techniques are able to predict future situations (based on past data) to a certain extent (probabilities). Another bias of the authors is their tendency to reject complex models simply because…they are complex. Although I know the famous Occam’s razor, advanced and complex techniques such as Support Vector Machine (SVM) have proven their efficiency in several applications. To conclude, whether you agree or not with the authors, Dance with Chance open your eyes on the illusion of control and thus on the limitations of predictive algorithms.