Tuesday, December 30, 2008

Data Trap: Extrapolating from a Small Sample

The press release says “The nation's small businesses own 93% of all "toxic" mortgages and are at risk of defaulting on their loans/payments.” The story is picked up by Barron's and blogs (Mr. Mortgage 12/16/08 and The Mortgage Lender Implode-O-Meter). But, what is this assertion based on?

Felix Salmon (Market Movers) does a great job skewering the survey used as the basis for the release, which contains too many errors to list here. The problem pertinent to this post is the conclusion is derived by extrapolating 1,687 survey responses to a base of 16.2 million small business owners.

Now, it’s possible to obtain good survey results from small samples when you ask clearly defined unambiguous questions and you have a lot of information on your sample and population. There is no way you are going to get good results on who holds what kinds of mortgages and their risk of default with a population as diverse as small business owners.

I’ve previously posted on the tendency of lenders to make unsupported decisions on asset quality based on prior but limited bad experiences. Those bad decisions are based on biases rather than poor survey design, but the effect is the same: an incorrect conclusion.