Why The Press Is Getting The Wrong Message Out Of The 'Nate Silver Walloped The Pundits' Story
from the small-sample-sizes dept
Let me start off by saying that I've been a longterm Nate Silver fan, back before he was the "fivethirtyeight" guy, and when he was just some random guy whose statistical models were helping my fantasy baseball team kick ass. And let me follow that up by noting that even more than being a Nate Silver fan, I'm a huge fan of statistics in general. I think that statistics should be a required class in school and that a combination of statistics and economics (the two go hand in hand) literacy (or lack thereof) is a major problem today, leading to numerous bad policy decisions. Finally, I've never been a fan (at all) of political punditry that focuses on the "horse race" aspect of politics. So, given all that, it has certainly been fun to follow the secondary storyline from last night -- which is how Nate Silver and his statistical genius "crushed" the pundits in predicting the election -- to the point that every single major press "pundit" was flat out wrong, and it looked like Silver had a perfect crystal ball. And, given how much Silver was attacked for being a "stats guy," (or for being biased, rather than neutral) you can certainly understand why it's tempting to wish he'd do something like Whitney McNamara's mock blog post:
Of course, if you look at what's happened to baseball since "Moneyball" and the success of the first statistical analysis guys, it should be a reminder that statistical prognostication is still about the probabilities -- and not about true predictions. And this is where the "suddenly-in-awe" pundits are still getting confused. They seem to think that Silver or other statistical modelers suddenly have a magic crystal ball with which they can predict the future. But probabilities and predictions are different, and Silver himself would likely admit (and, actually, did admit) that when you're dealing in probabilities, you're still going to be completely wrong some percentage of the time (he can even tell you what percentage of the time!) Even if the probabilities show a 90% likelihood that a certain event will happen, it still means that one time out of 10, you're going to be wrong.
Unfortunately, our brains don't deal that well with probabilities. We don't think in probabilities. Because we're dealing with a (mostly) binary situation, we assume that as soon as the probabilities tilt in our favor, it means that a "win" is somehow assured, and mentally, the probabilities turn into a prediction. It's very, very difficult for our brains not to think that way.
So I'm thrilled to see statistical analysis "win" over the moronic pundit-class who thinks that "storylines" or "momentum" (or, um, the ultimate in believing in anecdotes over data, "my friends see more yard signs" for one candidate) are valid methods for prognosticating. But it seems that the press, by going on to insist that Silver and his ilk are the new magic prognosticators, are missing the point just as much as those who thought the election could be predicted by political pundits.
Statistics is a tool for highlighting the probabilities. I'm sure that Nate Silver clones are going to be appearing a lot more on TV during the next major election cycles -- and I think that's a step forward. But now it seems like some people are expecting Silver and other stats guys to be right every time. And that's going to lead to backlash, just as the "failure" of Moneyball-type analysis to always get it exactly right resulted in some backlash in baseball. There will be data analysis in future election cycles -- likely from Silver himself -- that is wrong. That's the nature of probabilities. It will happen. And, unfortunately, people will then suddenly go back to arguing the opposite: that the stats geeks were "wrong."
But, as they say in the stats world, these are small sample size issues. Believing that statistical analysis is a perfect tool for predictions based on a single election is almost (though not quite) as weak as some of the traditional political punditry methods for predictions.
Hopefully, as with baseball, after a few years, the whole idea that these are entirely separate worlds will melt away. In baseball, every team now uses detailed statistical analysis as a tool, and most seem to understand that it suggests probabilities that help them find underexploited opportunities. But no one relies on it as a crystal ball that predicts the absolute future. Hopefully we'll reach that same sort of equilibrium in political analysis as well.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: elections, moneyball, nate silver, politics, predictions, press, probability, pundits, statistics
Reader Comments
Subscribe: RSS
View by: Time | Thread
And hopefully we'll reach a point...
Nah, never happen.But we can still hope.
Good wish though Mike
[ link to this | view in thread ]
What I wish people would learn from Mr. Silver
Nate Silver knows his stuff, but it's a shame people only listen to his predictions. As you imply, it's even more interesting is when he talks about statistical analysis itself. We can all learn a lot that is directly applicable to our everyday lives from him.
A short example is an interview I heard with him where the interviewer said something about how Nate is claiming Obama will win. His response was on the nose: that he wasn't claiming any such thing. Obama obviously could lose. He's just quantifying the odds. In other words, if Obama lost, it wouldn't mean his predictions are wrong, only that the less likely outcome happened.
[ link to this | view in thread ]
It's also very rarely wrong.
If people understood it, they'd understand why casinos never lose, and why insurance isn't a ripoff.
They'd understand that just because your uncle smoked a pack a day until he died at age 95 it doesn't mean tobacco doesn't cause cancer.
They'd also understand why Nate Silver is right.
[ link to this | view in thread ]
But yes: that Nate Silver is simply trying to present the results of analyzing a large amount of data, and then trying to offer some context around that seems to be getting lost pretty quickly in the "50 for 50" uproar.
As was pointed out above, Silver wasn't every saying that Obama was going to win, he was saying that the available data indicated that Obama had a higher probability of winning. Silver could have been entirely correct in his analysis and still seen a Romney win.
Also: two minutes after posting that I realized that I'd missed the obvious headline -- I think this version is better, but the ball was already rolling on the other one: http://tumblr.absono.us/post/35203726587
[ link to this | view in thread ]
we must outlaw it to protect our multi-trillion dollar industry with 70 billion employees
[ link to this | view in thread ]
Me, I want to read those 6 comments.
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Bayesian
[ link to this | view in thread ]
Re: Me, I want to read those 6 comments.
[ link to this | view in thread ]
[ link to this | view in thread ]
[ link to this | view in thread ]
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
A perfect Nate Silver example of the 90%
[ link to this | view in thread ]
Imaging you are an excellent stock picker. When you are right, you make $50. But when you are wrong you lose $500. Your right 90% of the time. Over the course of 10 trades, your down $50.
Reverse the situation. Your a horrible stock picker, but manage your losses well. When your wrong, you lose $50. When your right, you make $500. Your right only 10% of the time, but over the course of 10 trades, your up $50.
You can be right 90% of the time and lose money. You can be wrong 90% of the time and make money. In these cases, probability is less important than managing risk.
Nates numbers are great as long as you accept that he will be wrong occasionally and that's part of the model and to be expected.
[ link to this | view in thread ]
Re:
I thought it was amazing that we all think logarithmically naturally when we are babies, but then learn to think linearly later on in life. We know the difference between a few and a lot, but when numbers start getting really large, we forget what that means. There has to be an evolutionary driver for this.
Of course, I disagree with Mike that Statistics should be taught along with Economics in school. It *should* be taught at home, and then reinforced at school. Teaching it as part of Sesame Street or something like that. And critical thinking and theory of knowledge. But then again, kids should be allowed to be kids too. Figuring how to teach while making it fun is the trick, and I don't think we've mastered that yet in our current old-folks run (grind the new teachers down) education system.
[ link to this | view in thread ]
Re: Bayesian
Is it easier to cope with this idea?: Given what we know before the election, and given a specific model of the system, then that model tells us there are 0.6 bits of information to be obtained from the election.
In contrast, the statement that the election “is a tossup” is an assertion that —using a different model— there is 1 bit of information in the actual election results.
[ link to this | view in thread ]
Like Predicting the Weather
Then comes science, statistics and analysis, with a healthy dose of supercomputers. Now you get as much data as you can, use the best models you can, run it all through computer simulations and you end up with a set of probable outcomes; an 80% (+/- 5%) chance of rain tomorrow, and a 60% (+/- 15%) chance of rain next week. At first this new technology is distrusted, but after a few successful predictions (particularly when the "wisemen" get it spectacularly wrong at the same time). Now the "wisemen" are out of a job (or have to move to increasingly gullible groups of people), and bitter about it. But people can judge for themselves whether or not to take their umbrella, and most can stay dry.
Unfortunately, we seem to still be at the narrative-based "wisemen" stage of politics (both forecasting results, and policy-making/voting); where what matters is the story, the emotional appeal, the personality. It would be nice if we could move on to the evidence-/logic-based stage, but while we may get there with forecasting, I have a feeling that evidence-based voting and policy-making is still a long way away...
[ link to this | view in thread ]
Re:
Let's say Silver thinks Obama has a 3/4 chance of winning. Scarborough puts it at 1/2. They both bet $A.
So if Silver is right, he expects to get back 3/4 x $2A = $1.5A.
If Scarborough is right, Silver expects to get back 1/2 x $2A = $A.
Whereas Scarborough expects to get back $0.5A and $A respectively.
However certain Silver is that he is right, he expects to get his money back, or make a profit. Whereas even if Scarborough is right, he still only expects to break even.
So, what does this tell us? The Scarborough doesn't seem to understand probability. Which suggests that Silver's prediction is probably the more reliable.
[Disclaimer: I have no idea who Scarborough is (although I've been there, and apparently there's a fair), what the bet actually was (although I vaguely remember reading something about it late last night), and know very little about gambling.]
[ link to this | view in thread ]
Re: Re:
It's like me saying there's a 100% chance of X, and you saying you think there's actually a 10% chance of not X, and me saying "Oh, yeah? why don't you put your money where your mouth is and bet $1000 that X won't happen." It makes me look stupid, since you never actually put your mouth there (even though it would be great odds for me).
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
Re: A perfect Nate Silver example of the 90%
In Montana, they gave the Democrat a 34% chance of getting re-elected, putting him 48.4%/49.9% behind. He got 48.46%, but the Libertarian knocked his opponent down to 44.90%. So that may be an oversight (perhaps of the original polls as well, not taking the Libertarian into account.
In North Dakota, they gave the Democrat an 8% chance of winning, with 5% in the polls. Yet she won by 1%. There it seems that the polls were mostly out.
So that's 2/33 wrong, or about 6% error rate. This is one of those interesting situations where the mistake actually helps support the prediction.
[ link to this | view in thread ]
Re: Re: Re:
[ link to this | view in thread ]
Re:
If they were to follow the rules that is
[ link to this | view in thread ]
Stats and Lies
IE, it is a tool, and all tools fail at some point - sometimes catastrophically!
[ link to this | view in thread ]
“We ran the election 66,000 times every night..."
Remember Obama's AMA on Reddit? The article suggests that this was driven from their data analysis. It showed that many of the people they're trying to reach was there...
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re:
You are assuming that the probability analysis takes no account of the scale of the gains or losses. In reality it would do so. The numbers ARE reliable. Your problem is that you are only looking at half of them.
[ link to this | view in thread ]
Re: Re: Re:
[ link to this | view in thread ]
[ link to this | view in thread ]
Re: A perfect Nate Silver example of the 90%
[ link to this | view in thread ]
The problem with ideal models.
Insurance is a ripoff because the process is corrupt. They have legions of lawyers trying to give them an excuse to not pay you. They might try to welsh even without the opinion of a coverage lawyer.
It's a conflict of interest problem inherent to "for profit" insurance.
The concept is not wrong, just problematic when allowed to be handled by Ferengis. It's the gap between ideal models and actual practice that turn people off of "policies lobbied for by economists".
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re: The problem with ideal models.
[ link to this | view in thread ]