Model Risk -- What? Where?

Even before the Credit Crisis, now evolving into the Great Depression, there was some discussion among the more angst-ridden members of the financial community that maybe the elaborate mathematical models used to justify some of the explosions in CDS and structured securities trading might be, just a little, how should we say, “in need of adjustment.” Since then, it has become established wisdom that the failure to take into account “model risk” led to the downfall of the markets. (Alternatively, the complete reliance that we placed on models to accurately predict and assess risk was misplaced.)

But this collapse of established wisdom is not a topic that’s being discussed in an organized fashion. Most in the risk business seem happy either providing some “tweaks” to some of the more esoteric mathematically-based models, or moving from a “normal distribution” to a “power law curve” distribution to (in theory) account for “fat tails” (an oft-heard phrase these days, personally, one I’m getting bored with – that’s not all that went wrong, folks, not by a long shot). If anyone at this point is not clear on what is the distinction between normal distributions, power law curves, and standard deviations is, let me know, I’ll be happy to add another posting about the topic. (There are quite a few out there, but some of them don’t shed a lot of light.)

Let’s look a little deeper into the issue. Let’s back up and start with a general description of a) what models are, and b) how they’re used in practice in the securities industry.

“Models,” as the term is generally used, refer to the spreadsheets on which approximations of reality are generally described in mathematical terms. In other words, an analyst, or team of analysts, say that “The situation which we describe in words as 'blah, blah, mortgage loan default rates, blah, blah, interest rate yield curves, blah, blah' can be described with the following equations used in this order and fashion to precisely reflect what the underlying data show to be the situation we just described in words.”

These souped up spreadsheets generally have lots of fancy graphs and some long equations, and the quants that create them throw around enough mathematical and statistical jargon when explaining their masterpiece to produce glazed eyes and an “OK, whatever you say” result from the business people (and risk managers) who aren’t proficient in the requisite math.

Unlike other similarly arcane areas, such as technology, senior management has been under the spell of the “math gang” (at least in part because no one ever wants to admit that they can’t understand something presented to them by someone lower down in the hierarchy – though, in actuality, given the past twenty years of scientific math flooding Wall Street, it would appear that a Ph.D. in advanced physics and familiarity with the behavior of non-linear systems is just another check box on every CEO’s to-do list) – and in fairness, the "math gang" and their models have made a lot of money for a lot of people. What they've never been able to demonstrably do, however, is describe reality.

For years, people have been explaining away the variances between observations and model predictions with all kinds of sophistry and semantics, while carefully avoiding looking at the house of cards that’s become obvious from the disparity between the numbers and what most of us consider real life.

Let’s take a simple example.

Diversification is the concept that holding a portfolio of unrelated assets reduces risk (a portfolio of equity, real estate and cash, for example, diversifies risk because the assets generally behave differently in the same environment). This is a core concept of Modern Portfolio Theory (MPT), and plays a prominent role in all institutional asset allocation approaches. But MPT is based on a historical relationship between asset classes as observed in a fairly short time period (20-40 years). The historical asset classes are, by the way, are a whole lot simpler in concept than what we see live in today’s market. The entire theory of diversification pre-dates the explosive growth in derivative markets, for example.

Beginning in the summer of 2008, and culminating in September 2008 (also known as MAD-FV (Mutually Assured Destruction, Financial Version)), the historical relationships between asset classes all went out the window. In somewhat technical terms, the correlations between all asset classes went to 1 – meaning they were all moving together (there was no differentiation). Today (18 months later) the correlations have reverted a bit – to ranges of 0.7 or 0.8, depending on who you talk to, but past historical norms have shown no indication of reasserting themselves.

One would think that this disruption of the core definition of diversification would have some repercussions, or at least generate a lot of discussion - especially from those folks who provide us advice on how to invest in our 401ks. But I’ve seen nothing at all in the popular press about this. Please point out any examples you may find.

I was at a GARP (Global Association of Risk Professionals) conference in NYC a couple of weeks ago and asked a panelist about the shift in asset correlations. The bank risk officer replied that this disruption in MPT was one of his worst nightmares, in part because locating all of the spreadsheets throughout the bank that had models that were built in part on assumptions about asset class correlation was in itself almost impossible. And even if you find all of the places on all those spreadsheets, software, etc. where those assumptions have been integrated, how do you fix it? We don’t have an answer – period.

Seems to me that this could be a bit of a problem. Especially since asset class correlation is only one small subset of the “building blocks” used to create models -- models on which traders make trading decisions and on which investors make trading decisions. Including your retirement investing decisions and mine.

And we haven't started talking about VaR …