Allen Engelhardt, physicist and former quant, provides a thoughtful response to my post the other day, where I asked, “Do physicists and engineers get similar [statistical] training?”
“Of course we do”, says Allen. He makes the point, though, that statisticians and physicists have different world-views. A physicist will “(first) try to understand the model … He will run thought experiments. Then he might run some statistical tests against the data.” I agree that’s a different process than that for a statistician, where in practice the data often come first and then, after data cleaning and exploratory analysis, a model is refined.
I’d go so far as to say that even the word “model” means different things in the two disciplines. In physics a model is a “true” representation of a reality, invariant and concrete … at least until some errant data point is discovered and a new model can be proposed, tested, and accepted. I use “true” there in quotes to reflect that scientific process: Newton’s Three Laws were “true” reflections of the physical world until relativity called for a new “true” model. For a statistician in most disciplines, though, a model is merely a useful summarization of noisy data. It reflects not an invariant truth about the underlying process that generated data, but a tool for identifying important effects and, sometimes, to make predictions.
I think we’re both in agreement that the problem isn’t the models themselves, or even the estimates from the models, but how those estimates were used and — crucially — by whom. Says Allen: “We knew and understood that the models were not valid on the tails, but there was no volume of trading on the tails, so it wasn’t very interesting.” That’s fine and dandy for the high-volatility trading group, where only consequences were the daily ups and downs within that one group. The danger came only when that model became a component of a VaR statistic reported to the upper management from the risk group, when the nuance of “not valid in the tails” was lost when it came to assigning capital allocation.
Allen goes on to say “Society is often expected to pick up the bill for tail effects… the cost of prevention may be bigger than the cost of fixing it.” Perhaps. But I certainly hope it remains in a financial institutions selfish self-interest (and I mean that only in a positive way) to avoid bankruptcy. This takes us into a longer discussion about the conflict between an individual banker’s short-term self-interest, and the long-term best interests of a bank. (I think Surowiecki had a column on this topic recently, but I can’t find it now.). But if there’s any lesson to be learned here, I hope it results in practices that lead these banks to understand the limitations of their modeling practices so that such failures can be avoided next time.
