Complexity, Predictability, and Cascade Effects

Duncan Watts has a great piece in the The Boston Globe titled, “Too Complex to Exist.” I love the illustration:

Some excerpts:

ON AUG. 10, 1996, a single power line in western Oregon brushed a tree and shorted out, triggering a massive cascade of power outages that spread across the western United States. Frantic engineers watched helplessly as the crisis unfolded, leaving nearly 10 million people without electricity. Even after power was restored, they were unable to explain adequately why it had happened, or how they could prevent a similar cascade from happening again - which it did, in the Northeast on Aug. 14, 2003…

Traditionally, banks and other financial institutions have succeeded by managing risk, not avoiding it. But as the world has become increasingly connected, their task has become exponentially more difficult. To see why, it's helpful to think about power grids again: engineers can reliably assess the risk that any single power line or generator will fail under some given set of conditions; but once a cascade starts, it's difficult to know what those conditions will be - because they can change suddenly and dramatically depending on what else happens in the system. Correspondingly, in financial systems, risk managers are able to assess their own institutions' exposure, but only on the assumption that the rest of the world obeys certain conditions. In a crisis it is precisely these conditions that change in unpredictable ways.

In the article Watts proposes some regulatory steps to limit the complexity of financial systems. I am not optimistic; it is very hard to restrict activities until a problem is obvious (see my post “Rising Markets Create Lender Losses” for more on this). I think a more pragmatic route is for institutions to create firewalls within the organization so that the failure of one business line doesn’t take the whole institution down (e.g., AIG’s CDS operation pulling down the insurance business).

The Problem With Models

I’ve meant for a long time to post on the problems with using financial models, but there’s just too much to say, and too much that has already been said more clearly than I can say it. Here are some links on this topic:

Data series too short – for example, extreme economic conditions are not captured in the model. See Underestimating the tails, at  Revolutions.

Overreliance on past patterns – assuming the future will be like the past. See Maths and markets at FT.com.

Bad assumptions – for example, housing prices won’t fall. See Don’t Blame the Quants, Felix, at Falkenblog.

Network externalities – for example, failure of your counterparty’s counterparty was not considered in your model. See Andrew Haldane’s “Why Banks Failed the Stress Test” paper starting at page 9.

Failure to adjust models for evolving conditions – see John Kay, “Financial models are no excuse for resting your brain”.

Failure to properly account for low probability events – see Naked Capitalism, More on Global Alpha, Quant Woes, and Joe Nocera, “Risk Mismanagement”.

Extremely Improbable Events Happen All the Time

Risk managers assert in their defense that the current economic crisis was an unforeseeable, low probability event. From Andrew Haldane’s paper, Why Banks Failed the Stress Test:

Risk managers are of course known for their pessimistic streak. Back in August 2007, the Chief Financial Officer of Goldman Sachs, David Viniar, commented to the Financial Times:

“We are seeing things that were 25-standard deviation moves, several days in a row”

To provide some context, assuming a normal distribution, a 7.26-sigma daily loss would be expected to occur once every 13.7 billion or so years. That is roughly the estimated age of the universe. A 25-sigma event would be expected to occur once every 6 x 10124 lives of the universe. That is quite a lot of human histories.

How is it possible that extremely low probability events occur? The answer is that, while many events are highly probable over a short period of time, over longer periods events are extremely improbable. It is highly probable that when you go to bed tonight, you will get up in the morning from the same bed. But, think back to where you went to bed twenty years ago, and the events in your life that brought you to where you go to sleep now. How likely was it that you ended up where you are? That you have the job you have? That you have the spouse and kids you do?

From Carl Bialik’s The Numbers Guy blog:

We tend to fixate on those events that are memorable, after they happen. Peter H. Westfall, a statistician at Texas Tech University, notes that any given order of a shuffled 52-card deck has about a one in 10 to the 68th power probability of happening, including the sequence in which all 52 cards appear in order. “Everything we see has about a zero probability,” Westfall said. “Calculating these probabilities after the fact is kind of meaningless.”

The present we’re living has impossibly low odds of occurring.

Bank risk managers acted as though every tomorrow would be similar to the short term past, and didn’t account for less probable but still very possible outcomes (like house prices declining) which could rapidly create a much different environment in just a year or two (like the one we’re living in now).

Whose Error was the Housing Crisis?

Who is responsible for the housing crisis? Some candidates are borrowers, lenders, rating agencies, and securities investors.  Attempts to blame one party or another fail, because the crisis is the result of a combination of errors by different parties which all aligned. Think of a wedge of Swiss cheese; to see through it, all the holes must line up. This approach is explained in James Reason’s Human Error, and illustrated in a diagram from that book:

In the housing crisis, here are some errors which had to align to get to where we are today:

1) Borrowers took out loans they couldn’t afford

2) Lenders made loans to borrowers which the borrowers couldn’t afford

3) Ratings agencies rated securities comprised of these loans as safe

4) Security purchasers relied on the erroneous ratings and bought the securities

Any of these parties could have averted the crisis had they avoided their respective error.

I am not saying that every member of each class made their error; plenty of potential borrowers didn’t borrow, not every lender made bad loans, not every rating was bad, and not every investor bought bad securities. But, enough of each class made these mistakes to trigger the events leading to the current situation.

Also, I am not saying that individual actors didn’t benefit from their actions at the time – there were certainly some winners. And, looking at each individual decision made, it’s not clear that any of them were irrational at the time. These were errors in the sense that, in hindsight, collectively we would have been better off if people had acted differently.

In any complex system, it’s often more likely that a major breakdown is the result of an alignment of errors, rather than the failure of a single component.

Loan Underwriting, Financial Cycles, and Ponzi Financing

Loan underwriting of all types (consumer, residential mortgage, CRE) follows cycles. From Edward Leamer’s “Housing and the Business Cycle” paper:

Why do lenders “forget all about risk”? I’ve previously argued it has to do with certainty of outcomes and slow feedback loops (see here).

The Inevitability of Errors

Errors are inevitable – no matter what the stakes, no matter how much you practice, things are going to go wrong a certain percentage of the time in any complex task or decision. The New York Times has an article with an excellent example: basketball free throws.

There is nothing in sports as straightforward as a free throw; the equipment is always the same, the geometry is constant, and there is no defense interfering. The only variables are the player’s concentration and control over his or her body. And yet, at the highest level of the game, it goes wrong 25% of the time, year after year after year:

In the National Basketball Association, the average has been roughly 75 percent for more than 50 years. Players in college women’s basketball and the W.N.B.A. reached similar plateaus — about equal to the men — and stuck there.

The general expectation in sports is that performance improves over time. Future athletes will surely be faster, throw farther, jump higher. But free-throw shooting represents a stubbornly peculiar athletic endeavor. As a group, players have not gotten better. Nor have they become worse.

“It’s unbelievable,” Larry Wright, an adjunct professor of statistics at Columbia, said as he studied the year-by-year averages. “There’s almost no difference. Fifty years. This is mind-boggling.”

And it’s not like the stakes aren’t high:

Last season, Memphis was 38-2 despite making only 61 percent of its free throws, missing an average of nearly 10 a game. The Tigers lost the national championship game after missing 4 of 5 free throws in the final 72 seconds against Kansas, which had made a late 3-point shot to tie the game and won in overtime…About two-thirds of a winning team’s points in the final minute typically come from the free-throw line…

Obviously, we need to work to eliminate mistakes and design systems to minimize the chance of them occurring. But, a certain percentage of the time errors will happen. Learn what you can from them and move on.

Management, Feedback, and US Air Flight 1549

Via The Big Picture, an amazing animation with audio of the US Air Flight 1549 takeoff and landing.

It’s striking how the flight controllers’ understanding of the situation lags actual conditions. I think there’s a parallel with management and regulator understanding of what’s happening on the ground (or in the air, in this case) during rapidly changing conditions.