I have read read Black Swan and I think Taleb's central thesis is there are valid epistimic gaps in every model, and that suggest there is at least one family of hedge strategies where the goal is to think as creatively as possible about what we don't know, but what could be, so as to reframe that lack of knowledge as a plan to go tromping around in a jungle looking for holes. But to do so requires wide-ranging knowledge across many domains, which is hard to develop.
Unfortunately that is the more common case, and I agree on the author on that that most people take a binary stance on polls.
It should not be the case for regular readers of a website whose main topic is statistical analysis, although I imagine that just before a major election they would see a large number of non-regulars just to check the polls.
Bur I was just commenting that that the author is spot-on at least in that many readers do not take these explanations into account and they see just 28.6% of winning chance and they automatically convert it to 0% in their mind.
And the explanation is that they don't understand math and probability (or don't want to), whether the author likes it or not.
It makes no sense to me to hold 538 accountable for readers interpretations of the data. If I wander into some realm of science I'm unfamiliar with (say, quantum mechanics) and start making false inferences from it, does that mean the scientists were misleading me?
I also listen to their politics podcast, and they constantly talk about how to prevent people from rounding up an 80% chance to a 100% chance. One change they made this cycle was to try say 4 out-of 5 chance (which is mathematically the same) to convey both the lack of precision and to make it more intuitive. I think it helps. Personally, when I talked to people about election outcomes, I phrased it as "Based on what I know, I will be surprised-but-not-shocked if x happens, and shocked if y happens" to differentiate between, say, a 20% chance and a 1% chance.
https://fivethirtyeight.com/features/donald-trump-is-winning...
https://fivethirtyeight.com/features/donald-trump-is-the-wor...
> But it’s not how it worked for those skeptical forecasts about Trump’s chance of becoming the Republican nominee. Despite the lack of a model, we put his chances in percentage terms on a number of occasions.
So it would be more accurate to say that he was careful during the second half of the election, after making significant mistakes (both mathematical and ethical) during the first half.
If your model can help you predict how it's going to fail, I'm fairly impressed.
> The lesson, rather, is that Trump’s campaign will fail by one means or another.
> Donald Trump Is Winning The Polls — And Losing The Nomination
> Our emphatic prediction is simply that Trump will not win the nomination. It’s not even clear that he’s trying to do so.
https://fivethirtyeight.com/features/how-i-acted-like-a-pund...
> saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor
Something seems to have gone wrong at 538 after their stellar performance in 2012. In 2016 even their Congressional predictions were only 90% accurate. Or perhaps 2016 was simply different from past elections.
https://thehill.com/homenews/sunday-talk-shows/414759-nate-s...
> "So in the House we have Democrats with about a 4 in 5 chance of winning," Silver told ABC's "This Week." > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world."
Dinesh D'Souza jumps on him for hedging his bets: > And just like that, an 80 percent chance for Democratic takeover of the House goes to 50-50.
Nassim Taleb also jumps on him in the same way: > @DineshDSouza is right: you don't change a forecast from 80% to 50% under uncertainty. Second time klueless Nate Silver makes a mistake. > When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS. > 4- Things are actually worse: that @NateSilver538 doesn't get that if BOTH X & Non-X are "extremely possible = Probability converging to 50-50 is EXACTLY my problem w/his misunderstanding of probability in forecasting, & point of my paper. > 5- FOR THE RECORD I am not translating @DineshDSouza's point that it was 50-50: I ALSO understood ~50-50 DIRECTLY from Silver's "extremely" in the linked "The Hill".
The Twitter fight happened on Nov 4, if you want to go look.
In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down.
> This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting)
This isn't the public's fault, says the author, it's 538's fault for not clearly presenting the uncertainty in the prediction.
Additionally, the author writes, the 538 polling-based model swung too wildly in the 2016 election. In doing so, it failed to present a single prediction on which it could be judged. It also failed to acknowledge that unforeseen events could change the polls.
Personally, I'd argue that the swinginess of the conveyed the uncertainty of the polls. I spent the last month of 2016 citing Nate Silver to argue on Facebook and Reddit that people were overestimating the certainty of the election forecasts. Fifteen percent of the electorate was up for grabs with just two weeks left in the election! (about 7.5% undecided; and 7.5% saying they'd vote 3rd party)
> This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them.
Nate Silver is being blamed both for trying to convey to the public that his predictions involve uncertainty and for the public not perceiving the uncertainties in his predictions.
Silver's point is that their model results in a probability distribution of outcomes and, at the time he was commenting, in 10% of the cases Dems won 19 or fewer seats, resulting in a GOP majority, and in 10% of cases Dems won 51 or more seats, resulting in an overwhelming Dem majority.
That those extremes were approximately equally likely (in CDF terms) tails is willfully missing the point that the vast majority of the 80% of outcomes in the middle between those relatively fat tails involved Democrats winning, which is why the prediction was a 4 in 5 chance of Dems retaining control.
By missing that rather obvious point, D'Souza unsurprisingly comes off as a moron and Taleb also fails to cover himself in glory.
Just based on that snapshot my take is even more grim. Taleb surely knows how probability distributions work. The fact that he sneeringly dismissed basic reasoning like that in the political context he did suggests not cluelessness but malice.
One thing we do know is that Taleb is very smart and very good at making money. And right now there's a lot of money to be spent pandering to the dumb nihilistic and pseudo-intellectual wing of the political right; indeed, the same people that D'Souza has been pandering to for decades.
I'd never come across anyone using this phrase until Nate Silver did. Is it a US English thing?
It doesn’t mean that a broken heart will definitely kill you, just that there is definitely a chance that it will kill you.
I could totally imagine Nate trying to emphasise the idea that 20% is a much bigger probability than say 1% by using language like that, but put too much weight on it here leading to confusion.
What on earth is a professional like Silver doing using the word "extremely" in that context?
If I'd have written that kind of statement in school, the teacher would have crossed it out in red pen and told me to rephrase.
If I toss a coin, are the outcomes "heads" and "tails" both supposed to be "extremely possible", with "landing on its edge" being merely "unlikely"?
(In case you can't already tell) I'm with Taleb on this one.
Extemporizing while being interviewed on TV, presumably to stress the uncertainty in the prediction, is what he is doing, not writing an HN post, tweet or paper.
I suppose that whenever you are interviewed on national TV, you only say exactly what you mean, nothing more and nothing less?
As for the "50/50" here, I don't think this is meant as being exact numbers (after all, the whole point of the issue is that those exact numbers don't really tell you anything, if anything still is possible and any outcome can be justified later), but simply as the common usage of a phrase in the vernacular for "we don't know either way".
The whole point of probabilistic modeling is to replace absolute decisions like "right or wrong" by continuous weights on the possibilities. If you absolutely need a definite decision, you can sample a prediction according to the probability assigned by the model. If the true outcome is x and the model assigned it probability p, then that procedure is going to be wrong (1-p) of the time. You could define that number as the "wrongness" of the probabilistic model, as a continuous analog of the definite case.
The advantage of probabilistic modeling is that you can also ask how wrong the model expects to be and get a meaningful answer. If there are many possible outcomes and none of them very likely, any choice is going to be wrong a lot. But you should expect a good model to have a small difference between its expected and actual wrongness. One might call that value "honesty".
The whole point about the current topic as well as of my post: You missed by about a thousand miles. Please read it again. It's really pointless to argue about a strawman created by you. Your model is useless, that's the point! It makes no real(!) predictions - not usable for anything apart from blowing ever more hot air, and if it doesn't come to pass, you are never wrong because you left the door open by not actually saying anything in the first place.
Do you not understand that the guy/his company did nothing at all? And that giving some arbitrary probability was/is utterly devoid of any meaning (especially if you can't be wrong whatever the actual outcome)? They could have made any prediction at all, what difference would it have made? That is the value of that "work".
However, I realize there's people who like such meaningless drivel. It is a version of appearing to actually do something while not actually doing anything. You make it into the news but you can never be held accountable because whatever happens happens, you just helped create a few more entirely useless headlines (apart from helping with page views and ad impressions of course). It's actually quite ingenious to misuse actually useful tools like statistics.
"Possibility" is like "optimality" in the sense that they are binary attributes and thus don't admin grades. Something either is or isn't possible. Qualifying something as "more possible" is a mistake, and qualifying it as "extremely possible" is just nonsense.
A guy who lives from probabilistic analysis should now better.
I could understand people downvoting my comment (it may be interpreted as harsh even if that wasn't my intention) but... yours? Something weird is at play here.
The only way you could get "50-50" from that is if you read "the Democrats get 51 seats" as "the Democrats win the House", because without that inference, he didn't say anything about the chance that they win the House. He compared a very lopsided win to a slight loss, and those were equally likely.
I'm not arguing with you, since the one thing I know is that there's no surer way to making a mistake than jumping in confidently on probabilistic issues, especially issues of interpretation rather than of pure mathematics; but I think you must have misspoke, or I must have misunderstood. The quoted sentence says 'extremely probable' (an absolute condition), not 'equally likely' (a relative condition):
> "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world."
Indeed, despite quoting it in my response, I made exactly that mistake. Thanks!
Isn't that the very readership they capitalize on?
on edit: grammar was atrocious, still bad now.
> Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting).
Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. In fact, Silver is constantly trying to get across how likely < 50% probabilities are with his football analogies (e.g. "a team down by 3 at the half actually winning") to emphasize how likely something like "20%" actually is.
If anything, I love Silver's reality-based approach exactly because he does acknowledge that all he can do is offer a best estimate based on available evidence, instead of saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor.