News for hackers. Stuff that matters.
Re:Nate Silver correctly called 50 out of 50 states by ericdykstra on Wednesday November 07, 2012 @12:18AM (#4752246)
Nate Silver's calculations are certainly more sophisticated that a simple time-weighted average of legitimate polling numbers, but are they more accurate?

His baseball projection system, PECOTA, was extremely complex, but barely, barely outperformed a simple 3-year weighted average with an age component (Marcel), and some years was worse. Other projection systems that didn't take player comps into account were better overall than PECOTA (CHONE being the best, before the creator was hired away by a team).

Re:Nate Silver correctly called 50 out of 50 states by 3JPLW on Wednesday November 07, 2012 @12:23AM (#4752261)
The difference? Sample size. In baseball, everyone has the same data. It's all the data available. There's no sampling going on; it's a projection from past performance. Which, in many ways, is way more difficult.

His advantage in 538 is that he is using very sound statistical methods to combine multiple polls to increase the effective sample size and shrink confidence intervals. The complexities there are all related to increasing the individual pollster's historic performance and simulating the actual election.

Re:Nate Silver correctly called 50 out of 50 states by _delirium on Wednesday November 07, 2012 @03:25AM (#4752709)
I agree with that, but that's part of why I'm not convinced of his overall model's necessity: I think he gets the vast majority of his predictive power simply by being a good poll aggregator, who lays a reasonable statistical model over the polls.

His model has a bunch of other stuff too, though: estimates of how current jobs numbers effect the election, "state fundamentals" based on demographic variables fit to historical election data, etc. I'm not sure these things are adding much predictive power, especially given that they're generally fit to sparse, heterogeneous, and noisy data sources (e.g. the 11 Presidential elections 1968-2008).

On the other hand, one difficulty in evaluating it is that a good portion of the work his model is doing is not aimed at giving the point estimate, but the distribution and probabilities, to give an accurate probabilistic picture of the current state of the election at any given point, integrating various available data and historical trends. The goal, which is sensible, is to take a more rigorous approach to numbers you hear thrown around like, "when an incumbent is up by x% y weeks before the election, they win z% of the time". But even now in retrospect we can't easily evaluate how good those distributions were: we can see how well his point estimate fared, but with only one instance of how the election turned out, can't evaluate his overall picture's accuracy.

Re:Nate Silver correctly called 50 out of 50 states by waterlesscloud on Wednesday November 07, 2012 @03:38AM (#4752723)
I'm not sure his model adds much in itself, but...

I think he adds tremendous value in his write-ups. He explains what he's doing in pretty good detail, and provides some insight into what he believes it means. If you're seriously into it, you'll find places you disagree with him (same as it works in any other field when people are into it), but even if you're just casually interested you'll come away with some additional understanding. Or at the very least some additional questions of your own.

And he manages to write it in such a way that general audiences can pick up some of it, and come away with the impression that there's a lot more there. Not saying it's not a true impression, just that the fact that they get the impression at all is what matters.

Re:Nate Silver correctly called 50 out of 50 states by jimbokun on Wednesday November 07, 2012 @10:54AM (#4753786)
I believe Nate Silver is, by far, the best popularizer of statistical thinking in the U.S.
Re:Nate Silver correctly called 50 out of 50 states by spicyj on Wednesday November 07, 2012 @12:34AM (#4752299)
> to combine multiple polls to increase the effective sample size and shrink confidence intervals

How does this work? I was under the impression that to shrink the confidence interval by half you need 4x as many sample points, but there must be more sophisticated statistical methods at work here.

Re:Nate Silver correctly called 50 out of 50 states by 3JPLW on Wednesday November 07, 2012 @12:53AM (#4752359)
His models have a memory (increase sample size by using a handful of day or week old data) and aggregate as many as 5 to 8 independent polls per state plus as many as a dozen national polls that factor into his simulations. That's easily four to sixteen times the sample size.
Re:Nate Silver correctly called 50 out of 50 states by niggler on Wednesday November 07, 2012 @12:55AM (#4752367)
You can improve effective sample size by eliminating extraneous variables. One thing that Nate does is take advantage of implicit biases (for example, PPP polls lean slightly left)
Re:Nate Silver correctly called 50 out of 50 states by waterlesscloud on Wednesday November 07, 2012 @05:34AM (#4752947)
PPP seems to have predicted 332 EV and correctly picked the winner in every state they polled. They didn't poll some of the obvious states...
Re:Nate Silver correctly called 50 out of 50 states by kinofcain on Wednesday November 07, 2012 @01:05AM (#4752393)
And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%.

I haven't looked into Nate Silver's numbers, and certainly there would still be a reason for supporting him even if Romney had won tonight, but I think what most people are missing is that it's relatively straightforward to get pretty close to accurate, but incredibly difficult if not impossible to be completely accurate.

The predictions, as they get better, are going to asymptotically approach the actual results.

Re:Nate Silver correctly called 50 out of 50 states by Ecio78 on Wednesday November 07, 2012 @02:00AM (#4752536)
And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%

Yet at the end they decided not to implement it http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-...

Re:Nate Silver correctly called 50 out of 50 states by kaddar on Wednesday November 07, 2012 @09:05AM (#4753395)
Ugh I hate it when people make this claim or link this article. If you follow one more article link deep, they say implemented some of the earlier code that came out of it, code that provided the real business value consistently.

Do you know what caused a team to win?

They Realized (1) people rate movies higher on specific days ( why would you want to implement that into your estimator? And (2) they realized that all the movies that Netflix was asking about we're movies the user was willing to rate, and because users tended to rate movies they like, this biased the results, so basically, they didn't implement the hacking the test

Re:Nate Silver correctly called 50 out of 50 states by smoyer on Wednesday November 07, 2012 @05:41AM (#4752963)
They may not have implemented it, but they now know a) what's possible, b) one possible way to achieve the goal and c) how much effort is involved. I think the article misses the point as research often yields an answer that isn't a winning business proposition (at least at the moment). I'd love to know whether there were any actions taken based on the contest.
Re:Nate Silver correctly called 50 out of 50 states by DennisP on Wednesday November 07, 2012 @09:00AM (#4753383)
True, but other articles have said the main reason is that their business changed. The contest was based on DVD rentals, now they emphasize streaming, and that changes viewing patterns...people skip around, sample several movies before settling on one, watch their favorite parts, etc.
Re:Nate Silver correctly called 50 out of 50 states by phillc on Wednesday November 07, 2012 @09:05AM (#4753396)
And that they took some of the learnings during the contest and saw some decent gains.
Re:Nate Silver correctly called 50 out of 50 states by klapinat0r on Wednesday November 07, 2012 @09:09AM (#4753404)
For those interested in why and how, there's also http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-...
Re:Nate Silver correctly called 50 out of 50 states by gadders on Wednesday November 07, 2012 @05:45AM (#4752970)
They didn't implement it in the end? Doofuses (or doofii).
Re:Nate Silver correctly called 50 out of 50 states by Osmium on Wednesday November 07, 2012 @12:27AM (#4752278)
I think I speak for everyone when I say thank god politics is not like baseball :)

But seriously, congratulations to Nate Silver. I think this vindicates his method, at least for America. He was badly wrong with the last British election, whether because of bad polling data or an insufficient understanding of the system, so I was apprehensive going into this one. But it seems like, for America at least, he's cracked it. I think it's inevitable considering the absurd quantity of polling data available that someone would eventually.

Re:Nate Silver correctly called 50 out of 50 states by bgilroy26 on Wednesday November 07, 2012 @01:37AM (#4752490)
It shows how important it is to have experience and know your domain when you build a model and how understanding your subject matter is more important than cleverness.

Mr. Silver is one of the most clever people we have on the public stage, but that doesn't mean he has a silver bullet for everything. Rather, it enables him to work in a field that he knows very well and knock it out of the park.

Re:Nate Silver correctly called 50 out of 50 states by bitcartel on Wednesday November 07, 2012 @02:31AM (#4752599)
2010 British Election (Number of seats)

Silver predicted: Conservative 312, Labour 204, Libdem 103 Actual result : Conservative 307, Labour 258, Libdem 57

Re:Nate Silver correctly called 50 out of 50 states by tehwalrus on Wednesday November 07, 2012 @04:03AM (#4752778)
This may have been because a) Clegg's performance in the first debate led to a lot of polls where LD support was high, but ultimately this support melted away by the election and the history bias kept polls wrong until the last minute, or b) because there were lots of Polly Toynbees telling pollsters "I'm voting lib dem" to make sure they were in with a chance in the coverage, but actually intending to vote Labour (she actually advocated this tactic of lying to pollsters in the Guardian.)

Either way, it shows that statistical averages of what people say they'll do isn't always a good indicator of what they'll actually do when push comes to shove.

Disclaimer: as a (now rather embarrassed) Lib Dem, I'd quite like these predictions to be more accurate so I don't get my hopes up again.

Re:Nate Silver correctly called 50 out of 50 states by nl on Wednesday November 07, 2012 @08:37AM (#4753310)
I think your explanation is reasonable, but that's no excuse.

His US model makes a lot of allowances for things like how self-proclaimed independents are likely to vote.

It didn't work as well in the UK though.

Re:Nate Silver correctly called 50 out of 50 states by waterlesscloud on Thursday November 08, 2012 @04:25AM (#4757060)
I have no idea about the political culture in the UK. Is it possible that he just didn't have as much data to work with? In other words, a lot fewer relevant polls to feed into a model?
Re:Nate Silver correctly called 50 out of 50 states by tehwalrus on Thursday November 08, 2012 @09:30AM (#4757887)
we don't generally have very many regional polls, and certainly not constituency polls, that are available for aggregators (an internal poll in a key marginal, perhaps.) There's a lot less money spent on UK elections.
Re:Nate Silver correctly called 50 out of 50 states by tehwalrus on Wednesday November 07, 2012 @09:20AM (#4753446)
Indeed, I don't disagree. I'm just saying I think it may well be impossible to be completely accurate, or even remotely accurate, in crazy elections.
Re:Nate Silver correctly called 50 out of 50 states by Graham24 on Wednesday November 07, 2012 @05:33AM (#4752946)
103 for the libdems! that's one of the funniests things i've seen for ages.

how did he get it so wrong?

Re:Nate Silver correctly called 50 out of 50 states by shardling on Wednesday November 07, 2012 @07:57AM (#4753218)
In reading his analysis of the election here, he repeatedly mentions that having only two candidates makes things much more predictable.

In any case, we don't know if he got it "wrong" unless we know what % chance he gave this outcome. (He tries to account for the likelihood that polls are misleadingly biased, and that would correlate across multiple seats.)

Re:Nate Silver correctly called 50 out of 50 states by prof_hobart on Wednesday November 07, 2012 @07:26AM (#4753138)
Because lots of people lie to pollsters or change their mind based on the last thing they read in the paper?
Re:Nate Silver correctly called 50 out of 50 states by lifeisstillgood on Wednesday November 07, 2012 @12:58AM (#4752374)
>> I think I speak for everyone when I say thank god politics is not like baseball :)

politics is not nearly as important. :-)

Re:Nate Silver correctly called 50 out of 50 states by viviantan on Wednesday November 07, 2012 @01:09AM (#4752408)
Revenge of the Nerd. Sweet, sweet revenge :)
Re:Nate Silver correctly called 50 out of 50 states by clarky07 on Wednesday November 07, 2012 @01:01PM (#4754393)
wasn't he badly wrong in the 2010 midterms? I'm not convinced this means he cracked anything. RCP uses simple average of polls, and they got 50/51 correct with Florida being the only wrong if it goes dem (which it looks like but hasn't been called).
Re:Nate Silver correctly called 50 out of 50 states by lftl on Wednesday November 07, 2012 @02:11AM (#4752561)
Wes Colley, who runs one of the college football BCS ranking models, also did an electoral model. His model is much simpler, essentially taking the median poll result in each state.

http://www.colleyrankings.com/election2012/

Their predictions were basically the same as Sliver's, missing only on Florida (they manually broke a tie in the model on Virginia so count that however you'd like).

Re:Nate Silver correctly called 50 out of 50 states by cma on Wednesday November 07, 2012 @12:28AM (#4752280)
If you take all the simple weighted average with an X component and then selectively choose X after the fact, you can beat a lot of things.
Re:Nate Silver correctly called 50 out of 50 states by thewarrior on Wednesday November 07, 2012 @01:11AM (#4752413)
I've been reading Fooled by randomness , so i would ask , what proportion of his forecasting record can be attributed to just plain luck ?
Re:Nate Silver correctly called 50 out of 50 states by waterlesscloud on Wednesday November 07, 2012 @01:22AM (#4752443)
It's looking like the Real Clear Politics Poll average is going to be 49/50.

It was 48/50 in 2008 and 49/50 in 2004.

Re:Nate Silver correctly called 50 out of 50 states by foobarqux on Wednesday November 07, 2012 @06:53PM (#4755836)
Silver's model does seem unjustifiably complicated. For a simple model that managed to predict the electoral college see http://election.princeton.edu
Re:Nate Silver correctly called 50 out of 50 states by pbreit on Wednesday November 07, 2012 @01:34PM (#4754541)
Are evaluating baseball players and poll results even similar? One has a binary result, the other a spectrum. One is physically and mentally extremely difficult, the other a simple vote.
Re:Nate Silver correctly called 50 out of 50 states by daniel-cussen on Wednesday November 07, 2012 @02:15AM (#4752569)
Maybe he got better.