Right. It's like saying "Ha! You were wrong! You said there was a 83% chance I wouldn't roll a six, and I did"
Re:How Good Are FiveThirtyEight Forecasts?
by insickness on Friday April 26, 2019 @10:34AM
(#19757930)
More like saying, "We said there was a 17% chance that it could go the other way, so no matter what happens, we aren't wrong."
Re:How Good Are FiveThirtyEight Forecasts?
by b_tterc_p on Friday April 26, 2019 @10:46AM
(#19758063)
This is disingenuous. A prediction is a prediction. There is a world of difference between 51% certain and 99% certain. People excited that their data science models get good predictions on their test set without considering, say, the RoC curve are setting themselves up for failure.
If a model says 75% chance of X rather than Y, that doesn’t mean you should expect X. Depending on context, you might be better off flagging it as unclear or even safer to conclude a possible Y.
Testing calibration of many models across many different trials as a test of data scientist competence is a little strange, but if you want to answer the question of how valuable the predicted likelihood’s are, it’s a decent measure.
Re:How Good Are FiveThirtyEight Forecasts?
by Symmetry on Friday April 26, 2019 @10:41AM
(#19758004)
That's why it's important to do what 538 did here. Look at all the different places where they made 5 in 6 predictions ( and there are a lot of different races) and make sure that 5 in 6 of them happened.
Re:How Good Are FiveThirtyEight Forecasts?
by protomyth on Friday April 26, 2019 @10:34AM
(#19757933)
538's forecast was easily the most accurate for November 2016, by a decent margin.
No, Investor’s Business Daily was the most accurate.
Re:How Good Are FiveThirtyEight Forecasts?
by objektif on Friday April 26, 2019 @10:39AM
(#19757989)
If thats your understanding of making a prediction then he was accurate by your standards. You clearly did not watch the election night predictions. NYT live model was miles better than 538s.
https://fivethirtyeight.com/features/how-i-acted-like-a-pund...
Your link is from 2016 before the presidential election but after the primaries. Nate is admitting that he underestimated Trump in the primaries and talks about what happened. This does not seem particularly relevant to the subject of election night predictions.
Re:How Good Are FiveThirtyEight Forecasts?
by objektif on Friday April 26, 2019 @11:19AM
(#19758420)
This is regarding his bias. If anything the link is even more relevant. He repeated the similiar mistakes.
Re:How Good Are FiveThirtyEight Forecasts?
by pwinnski on Friday April 26, 2019 @11:25AM
(#19758485)
His main mistake as highlighted in that article is that they didn't use statistics for the nomination process, but guesswork. For the election proper, they did use statistics. So no, they did not repeat the mistake at all.
Re:How Good Are FiveThirtyEight Forecasts?
by notheruser on Friday April 26, 2019 @11:40AM
(#19758656)
His election night model wasn't good, but he isn't known and respected for his election night model, so who cares?
538's forecast was easily the most accurate for November 2016, by a decent margin. The majority of critics of 538 are "sore winners" who delight in their side having won despite being given a less than 50% chance, but as Silver pointed out at the time: his chances were similar to those in Russian Roulette, and most people consider that too risky to play.