Thursday, May 16, 2013

Nate Silver: What Big Data can't predict

http://tech.fortune.cnn.com/2013/04/26/nate-silver-what-big-data-cant-predict/

I blogged earlier on Silver's book - a wonderful candidate for ASET book club. The writing is crisp (the book short) and who knew an discussion of statistics could be fascinating.

This recent Fortune interview gives a small indication of Silver's approach:

Are there any questions out there that can't be answered using data and analytics?

So I think it all exists along a spectrum. It's important to know, too, that there's a difference between how good we are relative to our potential and how intrinsically predictable something might be. So for example if you look at baseball where analytics have come an awful long way, well it's still the case that the best baseball teams only win two-thirds of their games. The best hitters only get on base about 40% of the time. So it's still intrinsically unpredictable in a sense, but we have a good way of measuring and knowing what we know and what we don't know.

But there are a lot of fields where analytics have not come very far. I discuss earthquake forecasting in my book [http://www.amazon.com/The-Signal-Noise-Many-Predictions/dp/159420411X/ref=sr_1_1?ie=UTF8&qid=1367762705&sr=8-1&keywords=nate+silver+the+signal+and+the+noise] for instance, where people have been trying for centuries. We know something -- there are more earthquakes here in California than in New Jersey -- but the ability to anticipate a particular earthquake with any precision at a particular moment in time has not gone very well at all. Even economics -- when we try to do long-term economic forecasting, it has been pretty poor for the most part.

No comments:

Post a Comment