But the polls then and now weren’t just bad, they were shockingly bad and not representative of any real population that matters in an election. I’d expect something like this from junior engineers looking at metrics for the first time, but these are supposedly the most respective organizations in their field that have failed spectacularly for the second presidential election in a row. What value do they possibly serve?

Should Biden ultimately be declared the victor, I’m concerned that the public will never receive the postmortem on what went wrong and how it happened again that it deserves.

I really have no idea where this idea that the polls failed comes from. There were only two bad calls this cycle and every other outcome was within the margin of error. It was pretty much the same case in 2016 where people who had no idea what they were talking about suddenly decided they were certain polling failed because they are unable to grasp the concept of margin of error and sample size.

Polls can only guess about turnout and the try to work backwards from there. The turnout estimates were wrong but not shockingly so, and as a consequence a lot of polls ended up having the result be at the far end of their margin of error. Nothing went wrong. Polling is hard. Get over this idea that you can have some sort of certainty regarding an election until we actually hold the election.

If you dig into the the margins they aren't looking very good, even if they got a fair number of eventual winners correct.

Predicted vs Actual (FiveThirtyEight's averages vs NYTimes' current tally; Biden's margin is positive)

PA: +4.7% vs +.5% (-4.2%)

FL: +2.5 vs -3.4 (-5.9)

TX: -1.5 vs -5.9 (-4.4)

OH: -0.6 vs -8.1 (-7.5)

PA: +4.7 vs +0.5 (-4.2)

IA: -1.5 vs -8.2 (-6.7)

NC: +1.7 vs -1.4 (-3.1)

WI: +8.3 vs +0.6 (-7.7)

GA: +0.9 vs +0.1 (-.8)

MI: +8.0 vs +2.6 (-5.4)

AZ: +2.6 vs +0.6 (-2.0)

NV: +6.2 vs +2.0 (-4.2)

They pretty consistently overpredicted Biden's margin by about 4-7% in almost all of the swing states. Even if that's within, or close to, the margin of error; there's a systemic issue if it's happening in almost every state.

Edit: Added AZ and NV; and the difference

This is such a damning list.

At some point, reporting on polls when they have this bad of a track-record serves no journalistic purpose and just confuses the public. Like, the discussions people were having in the days before the election about Biden's strength barely resembles reality.

I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But they can only be as good as their sources, and when the sources are this terrible, they shouldn't be reported on this, and not given the statistical and scientific sheen of authenticity 538 gives them. Like, that site is starting to have a net negative influence on the world, and when that is the case with a journalistic institution, what are you even doing?

One of the things that's annoying about 538 is that they don't admit they're wrong. The excuse they keep repeating is that "1 in 10 chance Trump can win. That's about the same chance as rain in LA, but it does rain in LA!". 538 seems to have forgotten in their condenscencion is that we are not look at you to explain the odds. We are looking at you to provide an accurate probability estimate. Their defense is completely ridiculous. I follow betting odds which had Biden/Trump at 60/40 split before the election day - what's the point of polling when you can look at people putting money on the line, let the incentives work for you.

I'm not sure that's quite fair. They do write some excellent retrospectives and admit when they made mistakes, at least more so than almost any other news outlet I've seen:

https://fivethirtyeight.com/features/the-real-story-of-2016/

https://fivethirtyeight.com/features/how-i-acted-like-a-pund...

https://fivethirtyeight.com/features/what-i-got-wrong-in-201...

https://fivethirtyeight.com/videos/what-harry-got-wrong-in-2...

I have noticed them getting a bit more defensive recently which is irritating, but I think they are honest. There's only so much "garbage in garbage out" they can compensate for. If this ends up at +74 EV, which is where it looks to be heading, it'll be "Z=-.6" prediction (not actually normally distributed, but it's the easy calculation) prediction, meaning 27% of the outcomes were more Trump. It's not great, but I appreciate them giving a realistic model.

Thanks for these, I've been following it for last few months.

Nate Silver still makes fun of Betting odds: https://twitter.com/NateSilver538/status/1325186985988935680

I don't see him posting how the Bettfair was way more accurate than his polls.

I also get that Nate Silver doesnt poll himself, but he aggreates polling information, assign grades and mixes it into his formula. I just have zero trust in it. It's not like we get to test his predictions often. 1 sample size every 4 years.

Biden got out to around $4-$5 during election night. I wouldn’t put much weight in the betting when it appeared to fail to account for the expected composition of postal votes. Trump should have been at best a slight favorite before the effect of the postals started showing in the counts.

And why do you think they failed to provide an accurate probability estimate?

If I tell you that the odds of your dice roll being a 6 is only 16%, and you roll a 6, does that mean I failed to provide an accurate probability estimate?

The only way to judge the accuracy of a probabilistic model, is by quantifying the weighted error across numerous predictions. Not by looking at two yes/no outcomes. This is the exact point they are trying to get across, which you wrote off as annoying.

You do raise an interesting point about the betting markets - I too am very curious about whether they have a better track record than people like Nate Silver. Looking forward to someone analyzing their relative accuracy over a large sample of independent elections.

I saw predictit.org swing from pro Biden the morning of the election, to Trump in the afternoon, to Biden the next morning, so it seemed they were as confused as anyone.

They also didn't seem very aware that mail votes (leaning Biden) would likely be counted after in person votes (leaning Trump), which 538 had been predicting would cause a temporary pro-Trump lean for ages.

If you roll that same die 10 times and you get a 6 each time, then we can say that your probability estimate was wrong. But that's what happened with the polling of these states. The fact that the estimates consistently overestimated the Biden vote points towards a bias in the model used to estimate likely voters. If this were just a matter of the margin of error at work, each pull at the lever (each state) should be randomly distributed around the estimate. But that wasn't the case.

> If you roll that same die 10 times and you get a 6 each time, then we can say that your probability estimate was wrong. But that's what happened with the polling of these states.

The 538 model is explicitly based on the assumption that the state-polling errors are correlated to one another to some extent. Ie, if Trump performs better-than-expected in OH, he will likely perform better-than-expected in FL as well. This is why they rated Trump's chances as being 0.1, not 0.1^8. Given how polling works, this is exactly the right assumption to make.

Hence my earlier comment that if you want to evaluate the accuracy of 538 or any other model, you need to evaluate it across numerous different elections/events, over an extended period of time. Not a single day of elections in a single country.

This is kind of missing the point. The issue is the bias in the state-level polls. 538's model is to predict the winner of the election based on state polling data from other sources. The state level predictions are just averages of polls presumably weighted by quality. But this averaging cannot remove bias in the polls if that same bias is in many polls.

538 is a poll aggregator, not a pollster. If the polls are systemically wrong, which is what is being alleged above, there is <explicative> all that 538 can do to fix that.

Yes, I’ve been following 538 for a while now. They also grade each pollster.

The point is - 538’s input is a bunch of polls, their output is a prediction. Whether they do the polls or aggregate them is not relevant - they’re analysts whose job is to provide accurate estimates.

You can’t shift the blame on inaccurate underlying polls - Nate has time and again said, they look at many aspects in their estimates. Not just polls.

They look at many aspects at the beginning of the race. By the end these other aspects, the “fundamentals”, are purposefully dialed down to 0 and all that remains is an aggregate of polls. This is based on a theory that polls should be more accurate the closer you get to Election Day because voters have less time people have to change their minds. If those polls are systemically wrong, then there is nothing 538 can do to fix that; it’s a literal garbage in garbage out moment.

Looking at trends, which is easy to do on real clear politics, you could have seen the Senate was going to be extremely close, and the presidency, while not the blowout everyone expected thanks to 538, would still favor Biden. Here's one forecast that was arguably closer than 538: https://www.270towin.com/

The polls did tighten at the end, but you can't just look at a snapshot (even near the end) to account for the trend. In all of these states the polls narrowed in some, but the final results were within MoE:

FL: https://www.realclearpolitics.com/epolls/2020/president/fl/f...

PA: https://www.realclearpolitics.com/epolls/2020/president/pa/p...

NV: https://www.realclearpolitics.com/epolls/2020/president/nv/n...

AZ: https://www.realclearpolitics.com/epolls/2020/president/az/a...

GA: https://www.realclearpolitics.com/epolls/2020/president/ga/g...

NC: https://www.realclearpolitics.com/epolls/2020/president/nc/n...

In Iowa the Des Moines Register poll had Trump at +7, which is within MoE, the rest were garbage: https://www.realclearpolitics.com/epolls/2020/president/ia/i...

In Ohio, none of polls were within MoE, so all of the polls there were garbage: https://www.realclearpolitics.com/epolls/2020/president/oh/o...

For the Senate I think North Carolina was wrong everywhere I looked, but 75/25 for Democratic control at 538 was the most wrong I think: https://fivethirtyeight.com/features/final-2020-senate-forec... https://www.realclearpolitics.com/epolls/2020/senate/2020_el... https://www.270towin.com/2020-senate-election/consensus-2020...

You can go state by state[1] to see which polls were more/less accurate within MoE compared to the final result and the trends in each state. 538 had a better chance of Biden winning by 400+! EVs [2], than the 306 he's likely to win with. It's this distribution that could have been better at accounting for severe polling errors in some states.

538 did do CYA posts[3], but here while bringing forward the error from 2016 in Ohio seems right, it still projects a win of 335+ EVs. Optimistic for Biden is the kind way of saying what the final 538 projection were, severely more wrong than individual polls is more accurate. If their distribution in [2] was better, I would be more willing to give them a pass.

[1] click on the state: https://www.realclearpolitics.com/epolls/2020/president/2020...

[2] "Every outcome in our simulations" https://projects.fivethirtyeight.com/2020-election-forecast/

[3] "What if polls are as wrong as 2016? Biden still wins" https://fivethirtyeight.com/features/trump-can-still-win-but... https://fivethirtyeight.com/features/im-here-to-remind-you-t...

They provided the probability distribution. The fact that you can’t handle math and need some sort of absolute certainty for a future event is not 538’s problem.

Here is a guy who is fairly good at math saying that the polls did in fact fail:

https://statmodeling.stat.columbia.edu/2020/11/04/dont-kid-y...

https://statmodeling.stat.columbia.edu/2020/11/07/what-would...

That's a bit strong for what Gelman said. I'm a big fan of Gelman (and learned from his books!), but he specifically mentioned that both Gelman et al's Model and 538's Model did indeed capture the outcomes in their probability distributions, but that to improve performance going forward it was much better to predict closer to the median than closer to the tails. (And funny enough, Gelman gave 538 some grief earlier on making a model with very wide tails.) This is a nuanced but very fair criticism, and taking a Twitter-style summary of it I think is overly reductionist.

Ah yes. Mr 'let me tell you why Nate is wrong' Gelman, who is now Mr 'let me tell you why the fact that I missed bigger than Nate is not my fault and in fact is entirely the fault of these other people' Gelman. Forgive me if I find his excuses laughable, but I guess if it makes him feel better about himself we can humour him. He even manages to choke his first rant by missing once again on EV and vote percentages.

Whatever your complaint about Gelman is, I don't think "he can't handle math" is very convincing.

it's not one single election -- it's consistent failure over multiple state elections, by large margins, all in the same direction -- which falls beyond any reasonable probability

I don't think much of evgen's unreasonable personal attack. But 538 isn't necessarily claiming that the per-state error will be normally distributed around their predictions.

I don't know the specifics of their model, but probably they are claiming "with these polls, the probability of this outcome is...". The polls being consistently biased doesn't tell us much about 538s model. They said Biden would almost surely win and despite a massive surprise in favour of Trump, Biden won.

And even if Biden lost, 10% upsets in presidential are expected to happen once every 10 elections like this one.

> But 538 isn't necessarily claiming that the per-state error will be normally distributed around their predictions.

If per-state error isn't normally distributed, that's evidence of bias, or bad polling.

> If per-state error isn't normally distributed, that's evidence of bias, or bad polling.

No!

Assuming the per-state error would be normally distributed in some neutral world is making huge assumptions about the nature of the electorate, polling, and the correlations of errors between states, you can't do that! You would specifically /not/ expect per-state error to be evenly distributed because the nature of the error would have similar impacts on similar populations and there are similar populations of people that live in different states in differing numbers.

You should review the literature about the nature of the (fairly small) polling misses that impacted the swing states and thus disproportionately the outcome in the 2016 election. You will probably find it interesting.

Yes!

There are unavoidable, expected, sampling errors which are, by definition, random. That's why valid, trusted polls calculate a confidence interval instead of a single discrete result.

Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Maybe you relied on landlines only, maybe you spoke with too many men, or too many young people, asked bad questions, miscalculated "likely voter," whatever. Accurate, valid, trusted polls don't have these flaws, the ONLY errors are small, random, expected sampling errors.

> Accurate, valid, trusted polls don't have these flaw

Yes, they do. Because (among many other reasons) humans have a choice whether or not to respond, you can't do an ideal random sample subject to only sampling error for a poll. All polls have non-sampling error on top of sampling error, it is impossible not to.

when polls don't match up with reality, as they didn't in 2016, the pollsters have a responsibility to re-calibrate the way they conduct the poll. Ask different questions, find new ways of obtaining respondents from all demographics, adjust raw data, etc. A professional pollster doesn't just get to say, hey, some people didn't want to talk to me ¯\_(ツ)_/¯

> when polls don't match up with reality, as they didn't in 2016, the pollsters have a responsibility to re-calibrate the way they conduct the poll.

Pollsters do that for continuously, and there were definite recalibrations in the wake of 2016.

OTOH, the conditions which produce non-sampling errors aren't static, and it's impossible to reliably even measure the aggregate of non-sampling error in any particular event (because sampling error exists, and while it's statistical distribution can be computed the actual error attributable to it in a by particular event can't be, so you never no how much actual error is due to non-sampling error much less any particular source of non-sampling error.)

"There are unavoidable, expected, sampling errors which are, by definition, random."

This is false, if you think sampling errors are "by definition random" you don't understand polling.

Which is fine, just accept that you don't and dig into the literature or move on.

> That's why valid, trusted polls calculate a confidence interval instead of a single discrete result.

That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals.

> Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason.

Or the model was inaccurate. Perhaps the priors were too specific. Perhaps the data was missing, misrecorded, not tabulated properly, who knows. Again, the results fell within the CI of most models, the problem was simply that the result fell too close to the mean for most statisticians' comfort.

>That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals.

The CI is due to sampling error, not model error. If the error of the estimate is due to sampling error, the estimate should be randomly distributed about true value. When the estimate is consistently biased in one direction, that's modelling error, which the CI does not capture.

> If the error of the estimate is due to sampling error

What does "estimate" mean here? Gelman's model is a Bayesian one, and 538 uses a Markov Chain model. In these instances, what would the "estimate" be? In a frequentist model, yes, you come up with an ML (or MAP or such) estimate, and if the ML estimate is incorrect, then there probably is an issue with the model, but neither of these models use a single estimate. Bayesian methods are all about modelling a posterior, and so the CI is "just" finding which parts of the posterior centered around the median contain the area of your CI.

I'm not saying that there isn't model error or sampling error or both. I'm just saying we don't know what caused it yet.

Landed within the confidence interval? Are you kidding? CI is generally 2-4 points in these election polls.

- Texas: 538 said Trump +1.0, actually won by 6

- Ohio: Trump +0.4, won by 8

- Iowa: Trump +1.4, won by 8

- Florida: Trump -2.5, won by 3

- Penn.: Trump -4.6, lost by 0.5

- Nevada: Trump -4.9, lost by 2

- Wisconsin: Trump -7.9, lost by 0.6

https://en.wikipedia.org/wiki/Statewide_opinion_polling_for_...

https://www.nytimes.com/interactive/2020/11/03/us/elections/...

> Landed within the confidence interval? Are you kidding? CI is generally 2-4 points in these election polls.

The models and their data are public. The 538 model predicted an 80%CI of electoral votes for Biden as: 267-419, with the CI centered around 348.49 EVs. That means that Biden had an 80% chance of landing in the above confidence interval. Things seem to be shaking out to Biden winning with 297 EVs. Notice that this falls squarely within the CI of the model, but much further from the median of the CI than expected.

So yes, the results fell within the CI.

Drilling into Florida specifically (simply because I've been playing around with Florida's data), the 538 model predicts an 80%CI of Biden winning 47.55%-54.19% of the vote. Biden lost Florida, and received 47.8% of the vote. Again, note that this is on the left side of this CI but still within it. The 538 model was correct, the actual results just resided in its left tail.

Dude, you're gaslighting by using the national results as evidence instead of the individual states, which is what this has always been about since my original comment. Nearly every consequential state fell at, or beyond, the tail end of 538's confidence interval (BTW, who uses 80%? and not 90-95%?), on the same side. A bit closer to the mean in AZ and GA but same side, over-estimating Biden's margin of victory. Deny it all you want, gaslight, cover your eyes, whatever -- but clear, convincing, overwhelming evidence of a systematic flaw or bias in the underlying polls is right there in front of you.

Many political handicappers had predicted that the Democrats would pick up three to 15 seats, growing their 232-to-197 majority

https://www.washingtonpost.com/politics/house-races/2020/11/...

Entering Election Day, forecasters projected Democrats would gain House seats and challenge for the Senate majority.

https://www.cnbc.com/2020/11/05/2020-election-results-democr...

Most nonpartisan handicappers had long since predicted that Democrats were very likely to win the majority on November 3. "Democrats remain the clear favorites to take back the Senate with just days to go until Election Day," wrote the Cook Political Report's Senate editor Jessica Taylor on October 29.

https://www.cnn.com/2020/11/04/politics/2020-election-senate...

> Nearly every consequential state fell at, or beyond, the tail end of 538's confidence interval

While I haven't checked each and every individual state, I'm pretty sure they all fell within the CI. Tail end yes, but within the CI.

> (BTW, who uses 80%? and not 90-95%?)

... The left edge of the 80% CI shows a Biden loss. The point was 538's model was not any more confident than that about a Biden win. So yeah, not the highest confidence.

> Deny it all you want, gaslight, cover your eyes, whatever -- but clear, convincing, overwhelming evidence of a systematic flaw or bias in the underlying polls is right there in front of you.

Posting a bunch of media articles doesn't prove anything. I'm not saying there isn't systemic bias here, but your argument is simply that you wanted the polls to be more accurate and you wanted the media to write better articles about uncertainty. There's no rigorous definition of "systemic bias" here that I can even try to prove through data, all you've done is post links. You seem to be more angry at the media coverage than the actual model, but that's not the same as the model being incorrect.

Anyway I think there's no more for us to gain here by talking. Personally, I never trust the media on anything even somewhat mathematical. They can't even get pop science right, how can they get something as important as an election statistical model correct.

Not necessarily. Errors, like outcomes, are not independently distributed in US elections. Politics are intertwined and expecting errors and votes to be independent on a state (or even county) basis is overly simplistic. This is also what makes modelling US elections so difficult.

Sampling errors are random, and expected. Other types of misses are not simple "errors" but polling flaws, like sampling a non-representative group, ignoring non-responders or assuming they break the same as the responders, asking poorly-worded questions, etc.

Occasional flaws in polling is understandable and tolerated. But when those misses repeatedly line up the same way, and are rather sizeable, that's evidence of either systematic flaws, or outright bias.

I'm not sure what a "sampling error" is. To echo the sibling poster, per-state sentiment is not normally distributed. For example, we know Trump is more popular among white men than other demographics. This means that if we were to create a random variable that reflected the sentiment of white men throughout the US, we would (probably though I'd have to dig deeper into the data) presume to see a higher median vote count in this demographic. However, we cannot say that Trump's popularity in Massachusetts is independent from his popularity in New York, because his popularity in the white male demographic is the dependent variable between both random variables.

> I'm not sure what a "sampling error" is.

Perhaps you shouldn't be commenting on polling then.

I was discussing in good faith, so I'm not sure why you chose to be snarky. Let's clarify here, I'm not sure what "sampling error" in this case would be, such that it is distinct from electoral trends at large. The random variables in question _are_ demographic groups. How is it meaningful to discuss sampling error if your assumption is that state and county data is independently distributed? The poll data that Gelman et al used is public data, I urge you to take a look and work with it.

Yep. But 538 does not run polls.

The inputs it uses to spit out probabilities is known to be bad. Any scientist or researcher who claimed to get valid results from known bad inputs would be ridiculed.

To offer a concrete example here, survery respondents are often biased based on who actually sees and fills out a survey. A common technique used to overcome this non-representative sample is called post-stratification. There are, of course, limits to post stratification especially in instances of low sample sizes, but techniques to overcome issues with data are well known.

Science does not require an unbiased sample from a normal distribution to work. Bias is a technical term that the field of statistics is very comfortable working with. Scientists can also often get good results out of biased inputs.

538 has corrections for bias already. They seem to have worked in this instance - I repeat myself but: massive surprise, Biden still president.

> 538 has corrections for bias already. They seem to have worked in this instance

See https://news.ycombinator.com/item?id=25017810

You are pointing at evidence that 538 correctly called 11/12 races using statistics, and their confident call on a Biden president withstood a 4-7% swing (!!).

The existence of bias doesn't invalidate their predictions. Everyone knows that polls can be badly off target in a biased way - that isn't a new phenomenon.

When they talk about X% chance of Y being president they should be optimising to the outcome, not the margins.

I am questioning - why even bother?

It's not like we do elections every month to test out their probability distribution against empirical data. The distribution collapses into a binary outcome at the end.

I have a dice. I claim the distribution is of equal outcome for each side. Well...we don't get to test the dice more than once. 1 sample size does not prove that the 538's predictions were right (or wrong).

Thanks for assuming I can't do math, no way to argue with someone but I am actually pretty bad at it. :-)

Everyone is bad at probability and statistical distributions, not just you. The problem with modeling elections is that there are so few of them and the data is very noisy and until quite recently rather suspect. Let's not pretend that this was a normal election, either in the candidates running or in the manner in which the campaign and election was conducted.

As to the question of why bother, it is because bad polling is better than no polling at all. Campaigns are now multi billion dollar enterprises managing tens of thousands of temporary employees for the creation of a product that will only be sold once and in 18+ months from when they start the process. Any data is better than nothing.

The fact that the public has become obsessed with polls is probably due to the ongoing nationalization of politics.

> I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs.

But 538 chooses how to weight the polls. For example, they only gave Rasmussen (which was far closer on these) a C rating, preferring less accurate polls.

FWIW, their articles clearly lean left [1], IDK if that affects their analysis/forecasts.

[1] https://www.adfontesmedia.com/interactive-media-bias-chart-2... similar to WaPo

Boy, that rear view mirror makes criticism a whole bunch easier, doesn’t it? :)

I am reminded of all the times in the last four years (really my entire adult life) where during policy discussions of any magnitude the ultimate “kill” for an idea is “but it’s only popular with $small_number of the vote” or “but poll after poll shows it as a losing position.” It’s used as a cudgel to push out things that would actually make people feel their government is doing things with impunity. Donald Trump’s approval rating rarely if ever dipped below ~43% (with the usual and huge error bars) and consequently republicans reported they couldn’t go against him ever because of the polls. “Spending political capital” is a concept where a ruling party takes unpopular actions/enacts unpopular law because it’ll only push their approvals toward their long run averages.

I think this needs to be a serious topic of discussion. After this administration, “governing by polling firm” seems disingenuous at best and outright detrimental to all involved.

538 does make adjustments based on what they think are biases in individual polls. So they can't blame it all on their sources. They've taken on some responsibility to evaluate those sources and adjust their model accordingly. The polls had significant bias, and 538 failed to fully adjust for it.

One thing I found interesting about the final polling averages was that they were much more correct about Biden's share of the vote, even when they were very wrong about the margin itself (since these polls don't add to 100%, that's possible). See below for the 538 polling average vs. the current NYT tally (% for Biden):

PA: 50.2% vs 49.7%

FL: 49.1% vs 47.8%

TX: 47.4% vs 46.3%

OH: 46.8% vs 45.2%

IA: 46.3% vs 44.9%

NC: 48.9% vs 48.6%

WI: 52.1% vs 49.5%

MI: 51.2% vs 50.5%

GA: 48.5% vs 49.3%

AZ: 48.7% vs 48.9%

NV: 49.7% vs 49.9%

Most are within ~1% or so (some for Biden, some for Trump) with some outliers being Wisconsin (-2.6%), Ohio (-1.6%) and Iowa (-1.4%) in Trump's favor still not being so far off (relative to margins of error).

I'm not sure what that "missing" bit in the polls really means (undecided?) but it seems the issue was that that bit of the electorate ended up going entirely for Trump in many places.

(edit: had the PA 538 # wrong)

That's really interesting, I hadn't sliced it that way yet.

I know FiveThirtyEight allocates undecideds evenly - if 8% respond undecided they assume 4% will break Biden and 4% will break Trump. In one of the podcasts they discussed this assumption and it's what they have found to be the most accurate historically.

We might be seeing a shift towards that no longer being true - maybe 75% break red and 25% break blue now.

Edit: Also, It looks like Jorgensen (Libertarian) significantly underperformed her polling (~3% > ~1%) in the first few states I spot checked. If those voters broke for Trump, that makes up about 2% of the error. That'd take a lot more in depth checking though.

Probably worth noting that the asymmetric break of the undecideds in 2016 was what won the election for Trump. There were a lot of undecided voters and a lot of them decided for Trump late in the process so models that assumed a traditional 50-50 split were wrong.

It breaks against the least popular. Clinton was a known unpopular. Now that everyone knows Trump, he was the least popular.

Or they were decided and weren't forthright about it.

^ Exactly.

The polls were at least as wrong in 2016, even if that error was less significant this time (because the polled margin started out bigger).

Pollsters and poll-watchers downplaying this is not a good sign for future improvement.

EDIT: Overall though, I'm not sure that's a loss. People pay way more attention to polls than they should anyway.

There definitely seems to have been systemic error. Note though that many of these states are likely to rise by 0.5-1.0% as they finish counting ballots.

There are tons of sources of systematic error in a "strange year" election plus a "nontraditional incumbent".

One of the most plausible ones mentioned by Nate was the fact that people who wfh weré more likely to be reached by pollsters this year, and may skew D.

But you didn't actually show the margin of error. In any case, I think I'd take this argument more seriously in an article of its own, rather than on a back-and-forth forum like this. Not enough opportunities to request and provide context.

This exactly. If there's one thing all my physics and math teachers burned into my head, it's that a number without an error is meaningless.

Florida for examples may be off by 8 points, but 538 gave Trump a 1/3 chance of winning, so it was well within the error. What people don't realize is that the error on polls are pretty damn big, and elections in modern times have been extremely close. Like most of the swing states come down to under 100k votes / 1%. It's insane how close these rates are, and no polls will ever have any chance at predicting things like that.

At least in the circles I’m familiar with, there is open discussion about faking answers to pollsters because of how much the media is distrusted.

The problem is even these have narrowed as votes have counted. PA might be ~1 point victory. Many of these have shrunk. We can't really say what the 'actual' is until all the votes are counted.

And its actually much worse in most of the states that voted for Trump. IIRC MO was +8ish Trump in the polling and ended up +16 in the election

You wasted your efforts by comparing a point estimate of a probability distribution with the actual outcome of the single election day.

Not only that, 538 gave Trump a 1/3 chance of winning Florida, so it's not a surprise at all, and all things considered now that the votes are almost final, 1/10 chance for Trump to win seems reasonable. But as you mention, there's no way to figure out what the true probability was.

The bigger mistake here though is looking at the mean and ignoring the error on those values. 538 is fairly conservative in their calculations and their error bars are pretty big, so almost all of these results are well within their error bars.

Obviously, the focus is on the presidential polling, but I wonder if polling is off by the same amount when there are polls regarding what Americans want or think about any given topic.

I wonder if fear of being outed as a Trump supporter prevents pro-Trump people from responding to these polls, contributing to the skew towards Biden.

There's definitely a demographic who voted for Trump, but aren't open about it for social reasons.

That wouldn't explain why the polls were off in the Maine Senate race by 10 points though...

My issue with polling is that many races were well beyond the margin of error. The Senate polling in particular was bad this year.

Sara Gideon was favored to win the Maine race in the polling, because there hadn't been a single poll showing Collins in the lead since July. She lost her race by 9 points.

It's also strange to see the region makes a difference in the poll error. The polls in Minnesota were basically spot-on, but in Wisconsin (demographically very similar), the polling average was Biden +8, with one ABC news poll showing him +17, the kind of outlier result you'd expect with a +8 average. He's gonna win there by ~1 percentage point.

There's something wrong with how a lot of these pollsters determine samples, or how they judge someone's likeliness to vote.

It was both the sample (read actual reports from pollsters on how hard it is to get a sample these days... people DO NOT want to participate in polls) and an underestimate of the number of voters the Republicans would get to the polls. If your LV model is wrong you are really flying blind and for various reasons both parties activated a ton of voters this cycle so we had an electorate that no one was able to model well. The last time this large of an electorate turned out (in terms of percentage of eligible voters) they were deciding between William McKinley and William Jennings Bryan.

> It was both the sample (read actual reports from pollsters on how hard it is to get a sample these days... people DO NOT want to participate in polls) and an underestimate of the number of voters the Republicans would get to the polls.

Exactly, polling is very difficult, and getting even more so.

I was polled a few years ago by Gallup or Pew (or one of the other well known ones). The call was from an unknown number and I took it. No way I'd do that now with all the robocalls.

Is it possible that the projections themselves affected the outcome? If people saw that they had a comfortable lead, they wouldn't be as motivated to turnout.

Then the polling results might ironically be more accurate if people believed in them less.

Yes they can affect the outcome, but the opposite direction than you described. People who feel their candidate is doomed to lose don't turn out. People like to vote for a winner.

edit: To be clear, when polls overwhelmingly suggest a landslide, it suppresses votes from both sides. But a much higher proportion of the losing side will choose not to vote, thus inflating the gap.

it's not just polls that are wrong but likely this premature projection as well, tens of thousands of provisioned and military ballots are still not counted. Recounts are still to happen in most swing states. Lawsuits are pending.

I wouldn't be surprised Trump actually wins.

Everyone would be surprised if Trump wins, because at this point it is so mathematically improbable that even gun-shy decision desks at major news networks are able to make the call without fearing they will look like idiots. Recounts have never moved a state election more than a thousand votes that I am aware of and Trump is behind by tens of thousands in the states that matter. Lawsuits won't do anything, because no on has standing to prevent someone else from legitimately voting according to the rules in place at the time of election, nor of preventing someone else from counting those votes as legally mandated.

Trump lost, get over it.

> Lawsuits won't do anything

Lawsuits could definitely do something.

If there is fraud or misconduct, that can matter. Hell, the US spent two years investigating some Facebook ads and hackers.

But right now there's no clear indication that will in fact happen.

Elections are run by the states and so in almost all instances the federal courts will defer to the state courts when it comes to things like determination of fact. So far there has been no evidence of either fraud or misconduct and the thin claims put up so far have been laughed out of court. Lacking any claims of fraud or misconduct the only other option is to somehow be able to prove a miscount and since everyone learned their lesson with the hanging chads of the butterfly ballots this is exceedingly unlikely. With no claims to be made by anyone with standing the states are on a clock in terms of exercising their Article 2 powers.

On the first Monday after the second Wednesday in December (Dec 14) the electors are going to meet in the respective state capitals, whereupon they will cast their votes and attach six copies of their vote to six Certificates of Ascertainment which will go to the president of the Senate (Chuck Grassley), two to the national archivist (David Ferriero), and then one to their secretary of state and one to the chief justice of whatever federal district their state it in. Voila! Now at 12:01 on the 20th of January anyone, even you, could deliver the oath of office to Joe Biden and swear him in as the 46th president.

> Elections are run by the states and so in almost all instances the federal courts will defer to the state courts when it comes to things like determination of fact.

Yes.

It’s not just military ballots. I believe it’s also ex-pats. These may counterbalance each other.

well, as dems liked to say until this morning "count every vote".

Biden's margins are hair thin.

Biden is leading in most states by more votes than there are outstanding ballots; that does not really qualify as hair thin. The margin may be small in a few states, but it is large enough in a sufficient collection of states right now that while we keep counting every last vote we also know that Trump lost.

Want to bet?

If you look at RCP's polling average for Clinton in 2016, she outperformed that average for just about every race.

But, what the pollsters got crazy wrong, was Trumps polling numbers, by wide margins, far outside the margin of error.

I think they got it wrong this time too, and I think it all comes down to their methodology for actually getting a random sample of voters. Many polls are still married to live interviews and also to landline contact, and also tied to live interview polling - I think there is a partisan slant that they are not accounting for that includes a "propensity to answer a polling survey in the first place"

> I think there is a partisan slant that they are not accounting for that includes a "propensity to answer a polling survey in the first place"

They're well aware of that slant, and try their damndest to correct for it. But apparently they just don't have a good model of how large or how volatile that propensity attribute is, and how it relates to political leanings.

I was polled this year. One of the first questions I was asked is if I was answering the call on a landline or on a cell phone. They weight those differently. Also of course my basic demographic info (wage, race, marital status, education level, home ownership status, etc.)

I think your whole comment is correct, but instead of not accounting for the propensity factor, they do... just badly :)

I have no idea what the solution is to this. Maybe there isn't one, and instead we can get a silver-lining effect: If people trust the results of polls less, they're hopefully less likely to think the results are preordained. More voter turnout?

>I think your whole comment is correct, but instead of not accounting for the propensity factor, they do... just badly :)

In particular, Trump was extremely critical of polls and mail-in voting, and we have pretty much confirmed that his statements caused his supporters to avoid mail-in voting, so it's not too much of a leap to suggest the same thing happened with polls. Notably, this is a different effect from 2016, where IIRC Trump appealed to "non-likely" voters whose responses were inappropriately discounted.

> I have no idea what the solution is to this.

It is a solution that has its own problems, but mandatory voting would solve this problem. The polling failures are primarily in trying to weight the sample that you manage to capture in a way that reflects the actual turnout on election day. If everyone is forced to vote then you eliminate the RV/LV problem and pollsters would only need to make sure their sample was representative of the population at large.

Typical. Silent protest is the _most_ fundamental form of speech, how shocking that the extreme left wouldn't like it.

https://news.ycombinator.com/item?id=23548802

https://news.ycombinator.com/item?id=20141336

https://news.ycombinator.com/item?id=20117157

https://news.ycombinator.com/item?id=19751970

https://news.ycombinator.com/item?id=19743834

Are the Uncle Police OK with silent protest? http://v6y.net/unclepolice.mp4

Slapping +/- 5% on top of a prediction that is systematically off by ~5% isn't "in the margin of error", it's just simply off. For one prediction, two predictions, no problem. They fall somewhere in this range with probability distribution (usually a Bell curve around the mean). But for hundreds, if not thousands, of polls to be so systematically off, it is not "within the margin of error" or a statistical fluke. If it was, you would expect a normal distribution around the mean of the polls. The mean was completely off.

Put simply, it was sampling bias. Pollsters screwed up. They either sampled the wrong voters, the wrong areas, mispredicted who would turn out, or voters did not accurately report who they planned to vote for. Garbage in, garbage out.

You can try to correct for sampling bias, but if you are blind to it, as your comment pushes more people to be, then you will fail.

Right, but the job of 538 is exactly to deal with that, and they do take into account the fact that errors are correlated, so if one poll is off, chances are they all are. That's why their predictions are fairly conservative. They gave Florida a 1/3 chance for Trump and 1/10 for the presidency. Both those numbers make sense.

Now you can argue that there's little utility if the error bars are so big, and you may have a point. The bigger issue here is that modern elections have been extremely close, and unfortunately we cannot get polling accurate enough.

I know they were within the margin of error, but the polls were all systematically within the margin of error in the same direction. Did a single poll that was wrong predict at GOP win and end up a Democratic win?

That has happened in basically every election. Errors in polls tend to be correlated across states. If they were independent, 538 would have been predicting a ~0% chance of Trump winning instead of ~10%.

Yes.

I can’t find that via Google. Help me out here.

"poll predicted Trump win" brings back a number of results for me, including https://www.express.co.uk/news/world/1354506/us-election-202...

Ah sorry, I was thinking of down ballot races and polls that would have had enough reputation to make it into the Economist or 538 polling averages. I'll do some more searching.

The DI poll is basically a push poll with questions like "Who do you believe is telling the truth about alleged Biden family corruption?”

https://democracyinstitute.org/poll-donald-trump-set-to-win-...

One could argue the strategic mistakes of the Clinton campaign (ignoring the rust belt) and near fatal mistakes of the Biden campaign (which looks to be on issues swinging voters in battleground states) were driven by bad polling.

When I hear the conversation about "the polls were wrong" it's never been "the predictions were outside the error margin" so much as "voter sentiment was not accurately measured in regions where voters were most likely to flip and why they would do so."

The polls showed that Biden would win Florida by 2.2%, and in fact he lost by 3.3%. Is 5% within the normal error margin?

There is no reason that polls would be significantly different from the final vote results in most circumstances, if they are competently run.

https://projects.fivethirtyeight.com/polls/florida/

> There is no reason that polls would be significantly different from the final vote results in most circumstances, if they are competently run.

There is lots of reason.

If you could wave your magic wand and get a representative sample of voters in your poll you would be right, but you can't. Instead you do something like call people on a telephone, get a sample of people who both answer the telephone and who give answers you think indicate they are likely to vote. That sample isn't going to be at all representative of the voting population, so you take your best guess at what the actual voting population looks like, and how representative each of the people in your sample is of the voting population, and weight it accordingly.

Your best guess at the actual voting population is probably wrong. Your best guess at how representative each person in your sample is of the actual voting population is wrong. Your doing things like assuming that every <race> person with <education level> votes similarly regardless of whether or not they answer the telephone and respond to pollsters because you really just don't have a better option.

Then why are they repeated ad nauseam?

Polling is hard and the result are likely non-representative. Great! Then don’t display them to the public and have pollsters in interviews acting as if their insight is in any way valuable to most people. They’re either incompetently wrong or, worse, intentionally manipulative.

Just because something exists doesn’t mean it has value in the wrong context.

Just because something isn't perfect doesn't mean it has no value. Polls communicate experts best guess at how the vote is going to turn out based on the data they have collected. It's likely to be a lot more accurate than a non-experts guess, or even the guess of an expert without any data. It's also not even possible to share the raw data because of privacy concerns.

The forecasts you see on sites like 538 and the economist are even better, because it doesn't just communicate experts best guess (democrats win by +7), but experts best guess at how accurate that guess is too (something like democrats win by +7 +- 6, except in a lot more detail and nuance).

On HN the catchphrase for this is basically "don't let perfect be the enemy of good".

Note: I made up the numbers in this post.

@gpm what I’m arguing isn’t for replacing good with perfect. I’m saying these don’t make it to the level of good. They provide no relevant or valuable information to the public.

If weather forecasters were wrong nearly 100% of the time, would they provide value? If a hurricane was approaching and they couldn’t predict probable paths within any margin of error, would they provide any value.

If a weather forecaster can predict the temperature next week to within a few degrees and can tell you when they are sure it will rain, when they are sure it will be sunny, and provide a good guess as to rain or shine on most of the other days you would consider that person a good weather forecaster.

Do you expect the weather report for the next week to nail the temperature of every day next week to within 1 degree?

The pollsters can tell you if a hurricane is coming, they can tell you when the weather might shift significantly (albeit not be absolutely certain about which direction it is shifting) and they can give you a very good idea of what to expect even if they do not get the exact temp for each day correct.

really though, what tangible value do polls provide anyway? i get that the polling over the last few years hasn't been great, but i don't really understand why everyone's so upset about it. did anyone actually make or change decisions based on these polls? they're just predictions anyway.

for instance, in your example, a hurricane's path directly affects citizens of cities that lie on that path—do they need to prepare to leave their homes, etc.? but is anyone actually basing their decision to vote or who to vote for on the polls in such a way that it significantly affects the outcome?

> what tangible value do polls provide anyway

I live in Canada but am working for an American country. I am considering if/when I want to move to the US, they informed me about the likely outcome of this election, which influenced my plans a non trivial amount (not a huge amount either, covid dominates in the planning process, but a non-zero amount).

For many people it can influence how they vote. If you're in a state with a potential to change the outcome of the election you are more likely to vote "strategically", i.e. vote for one of the two most popular candidates instead of vote for a third party that you prefer. On the flip side if you're voting in a non-competitive race you are much more likely to vote for a candidate who is unlikely to win in order to better indicate your preferences (i.e. you should be more likely to vote libertarian/green/...).

For many people who wins an election affects there careers going forward, in the US in particular a large number of people are fired/hired based on the election. Even if you're not in that position your industry might get more or less government funding, if you're a government contractor the projects you are working on might or might not be in danger of getting cancelled, and so on and so forth. Having better information earlier makes it easier to plan your life.

The US election is one of the most significant worldwide events that happens every 4 years, the idea that being able to better predict it is not valuable is... insane.

you definitely make some good points. i realized a few things i didn't think about after posting as well.

i guess i was thinking more along the lines of, most of the time i personally don't think you should be choosing a presidential candidate based on predictions of who may or may not win. even some of your examples are more centered on the ultimate outcome of the election and how to plan for it; i really think people whose lives could be affected that drastically should be planning ahead for that situation anyway.

nonetheless, i agree they do have some value, and perhaps i should have clarified my line of thinking more clearly.

I think it's clear that people being polled are not representative of those who are voting. Trump has eother managed to appeal to people resistant to being polled or has managed to disengage people that are being polled. Whatever your views on the man, it's remarkable.

What's more surprising to me is that he can do it consistently and that no one has seemingly appealed to these people before. Given that these are almost election winning demographics, it's odd that no one has spent more time on understanding these people.

You obviously have no understanding of either polling or statistics. This is like saying that if computer programmers were not incompetent we would not have bugs in programs, but since they all obviously do not know what they are doing we have a modern society held together with chewing gum and string because programmers are too dumb to get it right.

What does "competent" mean if none is competent? Are engineers incompetent because they can't put a person on Mars?

Sure, if some engineers told you "we are putting this person on Mars with a 99.9 probability", and the rocket in fact reaches Venus, the engineers are incompetent.

Also, polling in most European countries doesn't go this far wrong, on almost all polls. So it's not like "accurate polling is impossible", it's just the polls in the US that seem to have some problem (though there have been some spectacular failures in other countries as well, which itself may lead to some thoughts on the value of polls in general).

> polling in most European countries doesn't go this far wrong, on almost all polls.

Brexit and UK elections would like to have a word with you...

Polling everywhere sucks. Pick a country and chances are it either has no polling to speak of or else it has the same problems.

YouGov are pretty good, mostly because they grew up in the Internet Age.

But I agree with your general point.

It's not within the normal margin of how off polling averages tend to be. (Though every once in a while you'll get an abnormal miss)

But it is definitely within the margin of error for many individual polls, which often have a MoE > 2.5%.

I think the main problem is that some forecasts gave an overwhelming amount of “hope” for certain races. This costs real money, as political operatives make decisions on where to spend money based on these forecasts. The closer a race is, the more likely more money will help shift the race.

Unfortunately you cannot just wish away the cause-effect here. Did Democrats piss away a hundred million in South Carolina on a race Jamie Harrison was never going to win, or did Jamie Harrison get within striking distance because of that money and was the investment in party infrastructure and training of lower-level party coordinators going to pay off over the next decade? Texas did not turn blue, but the effort spent on Beto's race and now on MJ Hagar may build down-ticket strength and knowledgeable people who may end up turning the state in a few more cycles.

When I grew up California and Virginia were red states, and I am sure people then also said money spent there was not going to move the needle and was a waste. When you have a billion+ to spend on races sometimes a little hope now pays off in a decade or two even if you lose this race.

California hasn't had 2 Republican Senators at the same time since 1971. California became Democrat when the Republican Party realigned to the Southern Strategy, not because of campaigning.

Prior to 1992 the only time California had two Democratic senators was post-Watergate and then prior to that single term for Tunney you have to go all the way back to 1863 to get two Dem senators. California had frequently elected Republican governors (including one who went on to become president), had house delegations that were majority Republican for stretches of time, and except for Johnson's win in 1964 CA was a solid batch of electors for the Republican presidential candidate up until Clinton.

CA was a Republican state from the late 19th century up until the 1990s.

> I really have no idea where this idea that the polls failed comes from.

For one, Andrew Gelman (professor of statistics and political science at Columbia University). Who knows more about the topic than anyone here in this HN discussion.

https://statmodeling.stat.columbia.edu/2020/11/04/dont-kid-y...

He may know a lot about the topic, but has little experience in actually modeling elections. He also has skin in the game and an incentive to make sure everyone knows that his miss was completely not his fault and was in fact due to failures by someone else, someone over there who you don't know, yes, she lives in Canada...

So what statistical mistakes did this lead to in his analysis?

If the error skews in one specific direction for all the polls, then I don't think you can just excuse the polls by saying they're in the margin of error.

This is some kind of non-trivial process problem.

It was very very bad up and down the ballot. No shit one of our clients their poll was - i'm rounding here because this is my real name - +15%. They barely won <1%.

I wish pollsters would move towards very very large digital samples. You can't get a 15 minute survey but head-to-head you can get a huge sample for not a lot of money. And don't try to weight it to what you THINK the turnout will be. Break out likely voter or not. Do not add any inference or opinion or 'math'

All digital surveys skew heavily to a younger demographic and heavily male. You have basically captured the political opinions of the fraction of the potential electorate least likely to vote. How exactly is that worthwhile?

I don't buy that, and even if I did the flip would be worse if you look phone contact rates.

I've seen some good very large digital surveys. We do some - geared towards measuring ad recall/lift - and I find them valuable, especially a giant sample on only two recall + head to head questions

You can always ask for gender/age and weight it but that's part of what I see as the problem. so many 'traditional' surveys i see make fairly large adjustments, lots of 'looking back' at past turnout and over fitting based on personal bias. Especially when weighting from very small sub-sample xtab e.g. hispanics. im not a pollster and that's just my still fairly-insider / polling adjacent insight

Another problem with digital surveys is most of the firms do opt-in panels, like having people register to take surveys and get paid on mechanical turk. and then they weight from there. I think this is a problem. The surveys we do run inside of mobile ads, similar to Google / FB Brand Lift surveys they go after those who saw the ads or a truly random sample, not a small biased group of survey panel

People in this thread might be interested in this podcast episode: https://www.iheart.com/podcast/105-stuff-you-should-know-269...

They discuss a lot of the things brought up here and go into details about the actual mechanics of calling people, etc. It's an easy listen.

I feel like everyone who's criticing these polls is forgetting one thing: COVID. These polls can only be as good as historical elections, and there really is not a lot of data for US elections during pandemics.

3-4 point polling errors are relatively normal, definitely not shocking. That's why the best forecast models still gave Trump a nonzero chance of winning even though the polls showed Biden up by 8+ points.

Polling is really hard. There are fundamental problems with it that are impossible to fully solve. It's frankly amazing that they get as close as they do.

Nate Silver gave Biden a 90% chance of winning, citing 3-4 tossup and 3-4 likely states. I don't see how his predictions don't look reasonable. Silver gave Trump a 27% last election - that's a very reachable percentage . Polls have considerable margin for error. It's standard for people to take a poll result as an absolute prediction and then rail against any error but that's not the fault of the poll itself. [2]

[1]https://projects.fivethirtyeight.com/2020-election-forecast/

[2]https://fivethirtyeight.com/features/what-pollsters-have-cha...

It might be representative... It's just that people didn't want to say they're voting for trump.

Biden was already ultimately declared the victor.

I think the big issue was the "margin of error" in many of these cases. For example, when Quinnipiac did a poll a few days before the election [1], they said Biden was winning Ohio by 4 points with a +-2.5 point margin of error... which happened to be very far off. And this wasn't isolated-–if memory serves me, Wisconsin was even further off?

[1] https://poll.qu.edu/florida/release-detail?ReleaseID=3683

I really want to know what went wrong with the polls (again). Shy Trump voter doesn’t explain how wrong the polls were about Graham or Collins, outside of even the margin of error.

Accurate polls are important to running campaigns and to legislative positions. Selzer got her polls right in Iowa and she posited that she knew the local electorate better than other pollsters so she weighted correctly. Maybe Trafalgar Group has a better method or maybe it just weights GOP voters more heavily (we don’t know, it won’t disclose its methods). Or maybe Trump just breaks polling and the polls will be fine next time. Whatever the reason, I’d like to know, but I’m not sure we ever will.

I grew up in a right wing province of Austria and saw that sort of wrong polling throughout my youth — which is why I wasn't surprised the slightest about things going as they did.

It boils down to: a fraction of the electorate will vote for $bigotcandidate but would never ever admit it to anyone outside the voting booth, because they know it is wrong. They will tell people they will vote for $notbigotcandidate, but vote for him nonetheless.

This fraction is not insignificant and you cannot easily represent it in polls without guessing the effect. In my home province it eas consistently around 10 to 15% over a decade.

I don’t see how that explains the Collins/Gideon race, but I’m not a Mainer. Anyone here from Maine can chime in why a Mainer would lie about voting for Gideon?

My same question. OK you can be claim to be a "good Republican" and vote Biden but (R) down ballot. Why were Senate and House races also wrong? Why did Lindsey Graham win by so much when the polls didn't give him that lead? In South Carolina, it's ok to say you'll vote for Graham.

Bigots turned out. And lean Collins.

The campaigns do not rely on the publicly available poll data for the reasons you pointed out - they need accurate polling in order to make serious decisions about resources. They pay polling companies to get accurate polls and they keep this data private.

The publicly available polls are done for free by polling companies or by news organizations. Unfortunately some of these companies apparently have an agenda to push. So they skew their sample away from what the likely voters are to get the result they want for propaganda purposes.

I think that the rise of Nate Silver has ironically made this situation worse. He advocates taking an average of polls to get a true view of the situation. So if you are skewing you poll to get a certain outcome and you know the poll will be averaged, you need to warp your results even further.

Ok, disclaimer time - I don't have any inside knowledge or real evidence of any of the above. It does seem to fit however. I can't explain why it seems all the distorting is to make the Dems lead seem bigger than it is - that is, why aren't there pollers on the other side of the aisle doing the same thing?

So you'll have to decide for yourself if there is any merit to the above theory.

Biden's campaign said their polling showed the race as closer than the public polling. It also doesn't do good to have too positive polling if it suppresses turnout. And it doesn't do the pollster's reputations any good to be consistently wrong.

I've heard several hypotheses, but I don't have anyway to test them. They all seem to have merit.

The issue with your assertion is that it doesn't account for all the times the polls are right. See most mid-term elections.

I think we have to seriously consider that Trump just breaks things that otherwise would be working within reason.

What you're saying is that any result with a non-zero chance of happening should be unsurprising to us, because it was still in the realm of possibility? If that's the case, what use are polls? What makes an election forecast based on polls more valuable than a coin flip?

Your coin flip analogy is spot on. But instead of a 1 in 2 probability, Trump had a 1 in 10 probability of winning. We knew this probability precisely because of polling data (and historical error).

This wasn’t some rare weird outcome. This was probably a standard deviation or so closer to Trump in a forecast. Perfectly normal thing to happen.

Sure, the model had value precisely because it said that even if Trump did better than the polls suggested, he would likely still lose due to the paths of victory Biden had. That said, the polls have shown, in many cases, poor predictive behavior of actual votes, and as such, we should not just brush this off by quoting the distribution of outcomes.

Polls are astrological race science, and the reason you'll never be able to convince anyone of this is adequately covered by the book When Prophecy Fails. I've come to think that their bias is only slightly ideological, but instead springs more from a self-serving impulse to mirror the current beliefs of the audience most likely to believe in polls i.e. people overly impressed by credentials. They enjoy the sheer volume of numbers, rather than their actual usefulness.

Alternate phrasings of identical questions asked in polls creates differences in outcome vastly larger than the effects they claim to detect. Somehow Nate Silver managed to get rich by selling synthetic CDOs of arbitrary polls (to people with his identical political outlook.)

The only real poll is to ask the same questions, in the same way, as the ballot sheet does. Ideally these should be asked as people are entering or leaving the voting booth, and people should be paid for their responses, in order not to bias yourself towards bored loudmouths with nothing to do. This would be good for accuracy, not for usefulness in anything but generating a suspicion of election fraud.

Instead, they're trying to figure out what percentage of black people go to the polls in Missouri, and pretending that the self-selected black voter has more in common politically with the general black phone answerer than the Missouri voter. Relying on that arbitrary assumption is the entire basis of their field.

edit: I haven't read https://site.pennpress.org/aha-2021/9780812250046/race-and-t... yet, but I heard an interview with the author around the time that it came out, and it's another hidden history of a field that has completely unearned legitimacy. "Political Science" might largely be considered an outgrowth of scientific racism.

.

Results fall on a spectrum from surprising to unsurprising based on the modeled prediction and confidence.

I'm on the nerd side too, but I was quite surprised. Sampling isn't the only source of error in election polling - people also change their minds. In this election, people decided who they were going to vote for historically early - the lowest portion of people in the history of exit polls indicated they decided in the week before the election. In theory, this effect should reduce the width of the probability distribution, and yet, the polls were off again, in the same direction as they are always off, by a little bit more than average.

A little scrutiny here is worthwhile. In 2016, for example, the polls were off in part because pollsters weighted the black demographic in proportion to their voting behavior in 2008 and 2012 (when a black candidate was running, boosting turnout). The actual 2016 black turnout looked more similar to 2004 or 2000 - other elections where a black candidate wasn't running. Though I haven't dived into the 2020 polling data, it wouldn't surprise me if a similar effect were at play.

It's also become noticeable harder to do polling. Once upon a time, everybody had a landline, so random dialing worked really well. Now, some people have more phones than others, and people are increasingly less willing to talk to pollsters, increasing the error rate.

These are all important things to talk about - when you dismiss it as "oh yeah, polls are never perfect," you prematurely shut down the conversation. Don't let perfect be the enemy of the good. The polls could be better and it's important to talk about how.

There is also "what polls directly measure" and "what we infer from them".

Besides the usual adjustments for likely voter models, we also do things like infer how people will vote in House and Senate elections based on Presidential polling, because we do a lot more Presidential polling than down ballot, because money.

So one early thing that seems to have happened this year (and again, we are still early in the process to be making final verdicts in "what happened" retrospectives) is that people split their ballot more than anticipated, which means that the inferences we were making about down ballot races based on Presidential performance were off.

So not quite a polling error, but an example of uncertainty that may not have been correctly taken into account in forecasting.

<< These are all important things to talk about - when you dismiss it as "oh yeah, polls are never perfect," you prematurely shut down the conversation. Don't let perfect be the enemy of the good. The polls could be better and it's important to talk about how.

I think the complaint is that here the polls were not just imperfect. They were at best, misleading raising questions over methodology ( lessons, apparently, not learned in 2016 ), wishful thinking, and their usefulness, or, at worst, attempts at influencing desired income.

I can absolutely agree that polling is not an exact science, but now it is twice in a row that polling science has grossly miscalculated the mood of the nation.

What % chance did the polls/forecast give the republicans gaining seats in the house (I actually don't know but from reading pundits it sounds like very little)? I think the interesting thing this cycle is betting markets were pretty much also at odds with pollsters/forecasts on Trump's chances, basically baking in a pro-trump (R?) polling error. Perhaps there really are issues with some of these forecasts / uncertainty estimates?

I think I only saw projections for Dems retaining control (which was high). I think GOP gaining seats was always in the cards.

I think it’s interesting too how some of the qualitative predictions (like Sabatos crystal ball) were pretty close to the outcome in the Pres race.

In just about every model, but let's use 538 as the canonical example, there was a reasonable chance for the Dems to lose seats but a very, very low chance they would lose control. We are looking at a two or three standard deviation variance from expectations for loss of control but losing seats was well within the range of outcomes that should have surprised no one.

It wasn't that little at all. Eyeballing the 538 graph it was about a 38% chance of Republicans gaining in the house? Over a one in three chance.

https://projects.fivethirtyeight.com/2020-election-forecast/...

Hmm looks like 548 gave a 3% chance of R control and betting markets gave about a 15% chance on election night:

https://www.predictit.org/markets/detail/4365/Which-party-wi...

You can also try to work the probability distribution from this betting market and I'm sure they were more favorable to R than he was:

https://www.predictit.org/markets/detail/6669/How-many-House...

That's a pretty big discrepancy regardless, did he ever give an explanation as to why he disagreed with betting markets so much? Seems like betting markets had a ~ one standard deviation pro Trump/R polling error baked into their pricing.

>why he disagreed with betting markets

Because betting markets are kind of dumb? Nate Silver has uttered those words a few times, anyway.

They have a pretty long track record of being predictive.

Not in this case, clearly.

Actually i think the number of seats gained by republicans will be pretty close to the P50 of the betting market probability distribution. They seem to have done better than 538 at least there. It's impossible to verify the overall Biden/Trump victory probability mass function but you could look at state by state races and see who did better.

I don't look through these very deeply but just as an example your link shows that the highest rated prediction on Nov 1st was 242-245 which is actually higher than 538's average.

Yeah I just looked at the distribution on close of Nov 2. The modal bucket was 246+, and the p50 if you convert to a quantitative variable was about 238, which is almost right in line with 538. Ultimately I think the huge difference with betting markets v. 538 for this market was they had higher variance and fatter tails, which could possibly point to a win for 538, EVEN if 538 were legitimately biased!

https://i.imgur.com/O7UegtQ.jpg

We'll simply never know without 1000 realizations of this election but next cycle I'd like Nate Silver to bet money whenever he finds mispriced events/odds and show how much money he makes to strengthen the case that his model is an improvement.

if you're a data nerd, how can you NOT be shocked by so many results falling far outside the confidence interval (often multiple sigmas), and all in the same direction?

Well this measures how many people are willing to admit they would vote for Trump which is not surprisingly a lower number than people who will vote for him in an unobserved secret vote.

Even if the pollsters are the most neutral nonjudgmental people on earth there will still be a fraction of the electorate that will (irrationally) shy away from telling them the truth.

This effect is certainly real.

Ummm, absolutely not. Pollsters universally stated they either had "adjusted their models" after being embarrassed in 2016, or that the "hidden Trump voter" was a myth (usually combined with "Trump fans won't shut up about how much they support him"). Poll results were repeatedly presented as forecasts of actual election results.

I grew up in a right wing Austrian province which over a decade consistently polled 10 to 15% wrong (in favour of the right wing candidate).

I know those people in person. They are people who will tell even their spouse they vote the left wing candidate, but after the 6th beer they will tell you how they really want the strong guy in charge (note: Hitler is from Austria, voting the rightwing guy can still be a taboo for certain people).

So these people do exist. If it is just them who make up the difference — no idea, but they exist.

I'm not saying such people don't exist -- I'm saying pollsters either claimed to have adjusted the models to account for this phenomenon, or the pollsters claimed they don't exist.

There certainly were some skeptics of the polls, but these skeptics were generally marginalized or dismissed.

The point is, most media outlets who reported poll results did so with the assumption that the poll results would reflect the actual results -- NOT that the polls results needed to be adjusted.

It's clear now that Shy Trump Voters existed. Some opinion pages trafficked in arguments against their existence. I think of them in terms that our Austrian friend described. Some pollsters were better at accounting for them. I avoid "most media outlets" language and try to look at the data.

I don't think they accounted for it. These are people you can't poll

There was one small polling company who claimed they had a methodology:

instead of asking directly who someone would vote for they asked a number of other questions first to warm them up etc.

They predicted way more Trump votes - and were summarily ridiculed by mainstream pollsters I think.

What would you expect of a pollster? "Sorry, we are currently unable to provide satisfyingly accurate polls so we close shop for this election?"

That's a rather defeatist attitude. Perhaps be a bit more creative in how you ask the questions? Find new ways to interact with a more diverse group of people? Take larger samples?

Further, the media could refuse to echo consistently unreliable poll results. Ultimately the media loses when people can't trust what's being reported.

Right, but given how infrequently elections occur, how do you receive the feedback loop to iterate on your models? The reason why statisticians and ML folk like to get huge datasets is that it helps them refine their models. Elections occur every 4 years and often reflect very different political circumstances from previous races. Perhaps if we had continuous polling, then pollsters and models could become more accurate, but as it stands now, most of these models are based on historical data.

Five Thirty Eights' "snake graph", which hasn't changed since before election day, accounted for poll error successfully and predicted exactly which states would go blue and which would go red, along with which several states were uncertain. Some of the uncertains went red, others blue. Florida was the only outlier in precisely how red it went.

I don't get why people care so much about predictions vs actual results in case of presidential elections in the US. The rules look like they were designed for maximum entertainment and sudden changes. The fact that you can win with less votes. When the other person already has 250 electors, but then on the last second you flip Florida or Pennsylvania with a few thousand votes and bam! you are the president!

That's why people love watching your elections so much. Or maybe also because of the fallout in their countries.

You're picking the easiest number to misread - the overall "chance of being elected", which doesn't really mean anything.

Much more interesting are the direct results predictions, especially per state where they matter most. Those are polls that give a particular result with a +- confidence interval. If the polls were to any extent competently done, you expect the result to be well within that margin of error.

The reality though has been that the results have been well outside the margin of error, sometimes multiple sigmas from the actual result - that is simply unacceptable for a poll. The only conclusion should be that polling data can be completely ignored, as it doesn't seem to have any clear relationship with the final results.

Individual polls were sometimes outside the margin, but collectively the polls were not far off the mark and collectively they were within the margin of error. This is why sites like 538 were able to model the various outcomes with some degree of certainty and ONCE AGAIN manage to get results within the range of expected probabilities. You really seem to have a hard time understanding both polling and probability and in particular the relationship between sample size and error. I suggest a bit more research.

Dude, I really don't get why you're being so gung ho on this. The errors on the individual polls were way out of line with the actual results.

When we regularise these results like 538 or the Economist did, we get slightly better results, but they were still not particularly accurate.

This is a real problem, and not one that we should be ignoring. I gave the polling companies a pass after 2016, but that this happened again is extremely concerning.

> This is why sites like 538 were able to model the various outcomes with some degree of certainty and ONCE AGAIN manage to get results within the range of expected probabilities.

But they were super close to the edges of the intervals. Like the most expected number of electoral votes for Biden was 380-400, which is not what happened.

> You really seem to have a hard time understanding both polling and probability and in particular the relationship between sample size and error. I suggest a bit more research.

Please don't be dismissive of other people. Many people (including people who do stuff like this every day) were very surprised by the results. We definitely shouldn't tar and feather the pollsters, but I think it would be great if we could get an understanding of why the polls appear to have been systematically biased.

I think it's more fundamental than numeracy. If the polls suggest there is a 2/3 probability you are going to win an election, but you lose, does that mean the polls were wrong? What if the probability was 9/10?

I think the problem is trying to make sense of outliers and uncertainty, and that, in an election with 150M+ variables, you will end up with a result that isn't quite what anyone predicted. In other words: it's not numeracy, it's epistemology.

Fivethirtyeight's polling averaging model underestimated Trump in all the swing states. Trump wine 2-3 states predicted for Biden, and 0 vice versa. States that Biden had a supposedly strong lead in turned out to be statistical ties in the official vote. The direction bias in the polls (about 20 of 25) is obvious, it's not symmetric random error.

Modeling correlated polling errors is basically what 538's models are all about. They model how far off the polls could end up being from the actual result based on past elections.

The polls are never 100% accurate. The pollsters have to estimate how many of each type of voter will actually show up to vote and what the response rate for each type of voter was. It is not an exact science.

To which you then have to ask: 1) were the discrepancies within the margin of error? and 2) to what extent are you justified in critiquing the polls based on a sample size of one?

I understand that the voting percentage we saw nationally was within the margin of error that 538 predicted, and that the specific configuration of electoral votes we saw was also one of the outcomes considered by the models.

Polls should say: “we’re 2/3 confident you will win” and not “there’s a 2/3 probability you will win”. The uncertainty is epistemological; it’s not describing a stochastic process.

Great comment, agreed. The wonderful books from Nassim Taleb ‘Black Swan’ & ‘Fooled by Randomness’ touch on this in a way that forever changed my thinking.

‘We just can’t predict’.

Couldn't the problem be that polling is just numeromancy dressed up in fancy math?

The problem that has appeared again is that without a representative sample of information gathered, it doesn't matter how fancy your models are, you will not have accurate results. And quantifying how unrepresentative your sample is basically impossible to figure out in this situation.

> quantifying how unrepresentative your sample is basically impossible

Yeah, so... numeromancy.

Maybe they should poll about what the person will vote, but rather what the persons opinions are in general and then estimate based on that who will they vote for.

I am betting Facebook or Google would be able to better calculate the outcome than any polling firm. Bring in all the data they collect on each internet user, I am sure they can model the likelihood of voting and where the user is leaning.

I am pretty certain Facebook has the data to crunch very accurate predictions in elections large and small.

Based on what? I think part of the problem with our politics these days is confusing social media engagement / metrics (specifically twitter) with national sentiment. Seems like there's a large mass of unobserved "silent" Americans out there.

Whats even more frustrating is recognizing the population difference in day of voters and mail in votes, expecting a "blue shift" as votes are counted, and then watching pundits chime in on Tuesday about surprising results.

As far as I can tell the most surprising result was Trump's popularity among Cuban Floridians.

Cubans are anti-Castro Republican. The young generation of pollsters forgot that Cubans aren't just "generic Hispanic".

Once Biden's team let Trump paint him as socialist I was not surprised to see the Cubanos go Trump (or the Venezuelans for the same reason) but I will be interested to see some of the breakdowns of the Puerto Rican vote in FL and some of the Central American Latino vote in Texas; I am betting there is going to be a reasonably large gender difference in some of those groups.

To be fair, we still don't know the loss of votes through mail-in voting. Mail-in voting is more convenient which would likely increase participation, but missing signatures, naked ballots, late or undelivered mail, etc can prevent those ballots from being counted.

When there is a major difference in voting patterns between the two political parties, then polls may not actually be inaccurate. In this case, Democrats chose mail-in voting and Republicans, by and large, chose in-person after being directed by the president.

Also, some percentage of mail-in ballots may not have reached their destination because of possible election interference through reduced post office efficiency--in particular, fewer mail sorting machines, lower hours, deprioritized mails, etc.

In short, the polls don't take into account election interference in the post office or unsuccessful delivery. If most people from both parties were voting in-person, maybe we might have had more parity between the polls and election results.

Everytime I read a comment like this I think of that line from Dumb and Dumber:

“What are the chances of a guy like you and a girl like me... ending up together?”

“Not good”

“Not good like one in a hundred?”

“I'd say more like one in a million.”

“So you're you’re saying there's a chance!”

I know enough about probability distributions and modeling to know you can improve your models to lower that uncertainty.

And sorry, you are the Lloyd Christmas in this one.

As an outsider all I can think is that this is an issue with the US media. The easy story is talking about the latest poll. The more difficult, but actually useful story, is focusing on the policies being put forward by each side.

>Forecasts give us probability distributions based on historical polling error data.

They would give you a probability distribution if they were truly random samples. But in reality, response rates are very low, and pollsters make up for this using demographic weighting and modeling - so the results don't actually represent any real population, but rather are extrapolated from people who respond onto a hypothetical population that the pollster thinks is likely to vote. This is why they are spectacularly and systematically off, more and more over time. It's not just variance.

> Forecasts give us probability distributions based on historical polling error data.

... and a lot of assumptions about the supposed underlying probability spaces.

You could also fit the same data with "people polled thought they could vote until their vote was preferentially suppressed". So the polls will never match outcomes. And that it happened in the same direction two cycles in a row leads credence.

(I do not know why the forecasts are wrong, but I don't see why Parent is at all justified in his statement).

Statistics can be pretty unintuitive. On top of that, most people have never learned any statistics.