That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model.

If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.

If they were wrong, but “less wrong” than all others, you should pick their model (unless you have an oracle, because the alternative - flipping a coin or “relying on your intuition” is rarely better).

Also, it doesnt make sense to look at a single prediction to evaluate a model.

Out of all the predictions they have made (did you look at individual state predictions?), how many were correct (and how confident were they?) - how many were wrong (and how close to 50% were they?).

That is how you evaluate a model (aka cross entropy)

It's not "wrong" to predict a low chance for something that eventually happens. Unlikely events can happen.

That isn't the criticism. The criticism is the appellation of it being "unlikely."

For example: anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. FiveThirtyEight had Hillary with 70%+ chance of winning both, which to me, shows a deep ignorance of actual cultural factors.

> anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan.

There was a very decent chance that Clinton could have won in 2016 (if any factor had gone slightly better for her), and if that had happened, nobody would be saying this now. This is literal hindsight bias.

Aren't you doing the exact same thing that you're accusing me of?

My view is simple: the media completely, totally got 2016 wrong, mostly for sociological reasons. The people making the predictions simply had a huge blind spot. Brexit is another similar situation. The fact that Hillary almost won or Brexit almost didn't happen isn't really the point, because both things were never expected to be even remotely that close. Had the predictions been "Pennsylvania will be close", it would be relevant, but those weren't the predictions.

It seems like you are conflating probabilities with absolute certainties. If I had a 1:10 probability of winning the lottery, I would probably take it. If I had a 1:20 probability of getting injured if I leave my house today, I’d stay home. If I did get out but didn’t get injured doesn’t mean the model was wrong.

If you have a die that rolls a one 1/6 of the time, do you consider the die wrong?

No, but I don't consider it useful toward predicting the outcome, which is of course what this is all about.

Say more about "but I don't consider it useful toward predicting the outcome"

How else would one predict the outcome of a die roll, specifically?

If there was sufficient data to assign a 0% or 100% probability to an event, that’s what a forecaster should do. If there isn’t sufficient data, then anyone who claims there is a sure thing is a charlatan.

If I tell you you're not likely to get two heads in a row, and you do, does that make me un-reliable?

It's unfortunate we can't just run the election again a few times, and actually find the rate at which Trump is elected given the polls.

And it's not empty signalling if 538 assigned Trump a higher chance of winning; they were pretty much the only ones saying he has a chance. That is why people think the models are useful.

If everyone was wrong, it is reasonable to believe a low probability event occurred. However, given the extent to which people predicted a Trump loss (say 1 in 20), which is significantly rarer, given that the event occurred it suggests the model that predicted a Trump win with the greatest probability to likely be a more accurate model.