They are not. The professor wanted some kind of statistically significant proof that they were any better than Paul, the octopus who successfully predicted an astonishingly high number of soccer matches by random chance [2].
Say what you will about 538, but at least they care about not using anecdotal or imagined evidence. At least they're rigorous with their methodology. They're also actually good at it, unlike most pundits [3].
[1]: https://gimletmedia.com/episode/58-earth-pony/
[2]: https://en.wikipedia.org/wiki/Paul_the_Octopus
[3]: https://en.wikipedia.org/wiki/FiveThirtyEight#Final_projecti...
Predicting elections is, for the most part, an easy task. Polls are usually good enough to get some picture of the election at hand. One major shortcoming of a primarily evidence-based approach is how to call races without much polling (small sample size). Silver has consistently refused to cover these races at all.
But Silver commits two sins that you ignore: he wants to show off his 'accuracy' and he wants to improve on the polls. These two problems kind of intertwine - the 538 'polls-plus,' which incorporates predictors Silver thinks are significant e.g. endorsements or Nate's gut, performs worse than the polls forecast, which is just a weighted average of recent state polls. What kind of rigorous methodology allows for two different predictions and then picking the best one, after the fact? What kind of rigorous methodology results in a prediction method that's shown to be worse than reading the newspaper, time and time again? Where's the evidence behind endorsements working this year?
As a side note, Philip Tetlock is the Wharton professor, and Dan Gardner (interviewed here) is some dude.
Silver wrote a faux mea culpa earlier this year about how he was wrong about Trump and Sanders and so on. My takeaway was that he apologized for inserting his own opinions where he claimed to use data. Isn't that what Diggler does?
Really you're overthinking it, and you shouldn't feel the need to protect Nate Silver et al, they're completely unaffected by Carl (afaik they haven't even acknowledged his "existence") so don't worry about it
[1] = http://www.cafe.com/breaking-missing-journalist-carl-diggler...
> Even when all of Silver’s models for a given race turn up wrong, it never seems to be FiveThirtyEight’s fault.
That is not at all a fair summary of 538's position. They always take responsibility when they come up short.
> When the site badly whiffed on last year’s British election, it was the pollsters who erred. (link to: http://fivethirtyeight.com/datalab/what-we-got-wrong-in-our-...)
The article they link to here doesn't shift blame to the pollsters at all. While they note that the polls were wrong, they take full responsibility for their bad predictions:
"It’s our job as forecasters to report predictions with accurate characterizations of uncertainty, and we failed to achieve that in this election. We are not trying to make excuses here; we are trying to understand what went wrong."
It's a little bit annoying seeing some people who are satirizing the media class unapologetically misrepresenting others in the same breath.
FiveThirtyEight's model could work without polling (it's not the only predictor they use), but they don't publish results until there are at least two polls conducted. I would call that a feature, not a bug.
Also, FiveThirtyEight correctly predicted the outcome of every primary race that it modeled, on both sides, except one (Michigan Democratic primary).
http://projects.fivethirtyeight.com/election-2016/primary-fo...
It's unfortunately burned into my mind, because of the 24 bets I made on the primaries this year, it's the only one I got wrong, but I lost $75 on that bet.
Primaries have to be harder because there is far less polling information compared to the general election. Yes, they were unable to predict the Trump victory, but if the get 90-100% of states correct in the general election, then I doubt that much has changed with how polling reflects the electorate.
I'm not a Trump supporter by any means, but 538 went way out of their way to ignore every poll that was put in front of them in favor of punditry.
The correct solution is just to move on. Too much of the media coverage of elections is about the horse race aspect, almost none of which is actually of positive benefit. Rather than spending so much time trying to guess winners, the media could spend more time and effort actually covering the actual issues of the election in a mature fashion, but it's been decades since they even tried.
Carl is a work of fiction, invented by a couple of journalists/friends as a parody of pundits: https://www.washingtonpost.com/posteverything/wp/2016/05/09/... - the story is hilarious. Watching this whole thing unfold on Twitter has been a total blast too (check out @carl_diggler), they never break character and have a lot of running gags (such as Carl's troubles in "Family Court"). Here's a wee excerpt from that WP link:
" Carl exists to satirize all that is vacuous, elitist and ridiculous about the media class. From his sycophantic love of candidates in uniform to his hatred of Bernie Bros, from his reverence for “the discourse” to his constant threats of suing the people who troll him on Twitter, Carl is predicated on being myopic, vain and — frankly — wrong.
But something funny happened along the way. Biederman and I, who are neither statisticians nor political scientists, started making educated guesses for our parody about the results of the primaries. And we were right. A lot.
We beat the hacks at their own game by predicting every Democratic winner on Super Tuesday. We told readers who would win in the unpredictable caucuses that FiveThirtyEight didn’t even try to forecast, such as those in Minnesota, Wyoming and even American Samoa. We called 19 out of the past 19 contests. FiveThirtyEight, whose model cannot work without polling, accurately predicted 13 "