In a Data-Swamped World, Connecting the Dots Is All Too Easy

An information-saturated society is going to notice plenty of weird correlations, like the Blade Runner curse or the unfortunate fate of American presidents elected in years that ended in a zero (for a long time beginning in 1860, all died in office). The more data we collect, the more patterns we see. Iran leads the world in nose-jobs per capita. Major wars have been preceded by stock-market plunges on the other side of the world. It’s increasingly easy to find unexpected alignments between facts.

The hard part, the work of science, is figuring out which patterns are simply accidents—the equivalent of a coin toss coming up heads 20 times in a row, which is both meaningless and inevitable. The presidential-election curse, for example, popped like a soap bubble after Ronald Reagan (elected 1980) lived through both his terms. On the other hand, maybe there’s some truth about society or human nature to be found in the Iranian penchant for plastic surgery.

We can’t know without some work: an explanation based on a correlation needs to do much more than line up the wheels. It needs to posit some underlying cause for the coincidence, and suggest a means for proving, or disproving, that this hidden cause is real.

I know in the finance world there are many “quants” who believe their market data provides insights into non-financial events. But without a logical and findable link between stock tables and, say, changes in tastes in movies, their claims sound like so much numerological wishful thinking.

John Casti, a mathematician who has written a lot of interesting books about the nature of knowledge, has apparently been persuaded by the quants. His latest book argues for Robert K. Prechter‘s theory that financial data is a window onto hidden laws that drive history—that, read correctly, the stock tables will tell you why, and sometimes when, wars will break out, fashions change, political movements rise and fall. As you can read here, I wasn’t convinced.

Berreby, D. (2010). On markets and collective mood Nature, 467 (7311), 31-31 DOI: 10.1038/467031a