If I'm understanding the article correctly, it's not talking about models being "correct," it's just talking about the probabilities of all the outcomes adding up to 100%, which means you can't place a bet that guarantees you make money regardless of the outcome.
In other words, offering a bet that was based on the claim that there is a 99% chance of rain today in the Atacama Desert, and a 1% chance of no rain there, is arbitrage-free. It's still clearly a poor probability model for one of the driest places on Earth, but it's arbitrage-free.
If e.g. I know that you'll have a probability below 40% at some point in the future, and I know you'll have a number above 41% at some point in the future, then it's trivial to create a strategy over time that is guaranteed to make money.
Taleb's claim, as I understand it, is that Silver is incorporating random noise into his model and so will exhibit such predictably swings, which enables arbitrage.
If I am not mistaken, an opportunity for arbitrage is not created merely by a certainty that they will change somehow; it would require that they change in a specific way, and deterministically so. Absent this, a bet would just be speculation, not arbitrage.
And this is precisely the complaint Taleb is making.
An arbitrage is just a strategy that's guaranteed to make money. If you know for sure that something will go up and down at different points you could make money risk free. Simply buy at any point below the current price and sell at any point above.
This is not arbitrage. Also this is precisely the kind of strategy Taleb would have argued against in his book about black swan events.
If someone was flipping a coin and raising their price when it landed heads and lowering it when tails, arbitraging using reversion to the means would be a perfectly legitimate arb strategy.
Taleb's argument is that Silver's estimates are based on random noise, and are thus vulnerable to this form of arbitrage.
https://en.m.wikipedia.org/wiki/Risk-neutral_measure
I’m not sure exactly how the arbitrage works, though.
But the argument in the paper seems to be not that there’s noise but that there’s too much of it. In fact, it assumes that the expected vote share over time is subject to Brownian motion with specific parameters (which is a rather flawed assumption), but then concludes that 538’s probability distribution is more swingy what he’d calculate from those parameters.
Predicting the result of a vote for a day when it's not going to be held is a form of entertainment that can't be verified. Is Silver claiming any more than that?
Each time the probability drops below 40% (in your example) it is likely because the fair odds swung in the latest polls. To attempt to take advantage of this is to ignore the recent evidence (conditional upon that evidence being quite different to the current concensus).
It's not at all unreasonable to do so, but isn't arbitrage. What if all polls started being increasingly conclusive leading up to election? 40, 39, ..., 0. No surprises and no arbitrage.
Perhaps another way to express this critique of Silver's methodology is that it lends too much weight on recent outlier observations? Sure if you knew this you could profit off these swings, but making the arbitrage argument doesn't add much to the discussion to my mind.
I'm having a fair bit of trouble picking up that from either Taleb's or Clayton's article, but let's say that it's a fair assessment.
Is the complaint that the observed variance is high enough to enable these arbitrages? So in order to make his model arbitrage-free, Silver would have to provide a spread rather than a point price, with his forecast as the midpoint of the spread, and the width of the spread wide enough that .5 is always inside the spread, thus making the forecast economically useless?
Or is it that Silver does not provide error bars that would estimate that uncertainty?
Or is it that because of intrinsic uncertainty in the election?
But complaining that a model is incorrect because it enables arbitrage is a standard complaint, it's how you can prove all non-Bayesian models are irrational. That doesn't mean all Bayesian ones automatically pass the test though.
Error bars and spreads are already outside the Bayesian paradigm, which demands a single number.
> It's how you can prove all non-Bayesian models are irrational
Can you expand or link?
> Error bars and spreads are already outside the Bayesian paradigm, which demands a single number
Not following at all. What do you mean?
>Not following at all. What do you mean?
One assumption is that given a bet, you must choose to take a side of it. If you won't bet for something at X%, you must bet against it at 1-X%.
When I think of Bayesian analysis, I always think of Bayesian models which attempt to model uncertainty and always have a posterior distribution (spread) of outcomes.
For example, in the Dutch Book page, instead of having a point-prior (P=0.51 (delta on P=0.51?)), each degree of belief could have some (different) distribution which could yield a posterior with a distribution and change the math here.
To say that the probability of it being true is a range is incoherent.
It's not incoherent to analyze the posterior distribution by reporting a credibility interval instead of a single number (for example the posterior median).
How well that models an actual election is debatable, of course, but I think it does model it to some degree. In reality, there are not two stages but multiple, and none of those stages are necessarily fundamentally stochastic; rather, you just need exponentially more information to predict one stage than to predict the previous one, and without that information you may as well treat it as stochastic. For example, if I’m about to flip a coin, a god with exact knowledge of the state of my body and brain, the air currents in the room, etc. might be able to predict how I’ll throw it and how it will fall, but mere mortals have to treat a coin flip as random. Similarly, a god with exact knowledge about the state of the universe might be able to predict an election result eons in advance… though quantum randomness might trip them up. Getting more down to earth, if you just could poll every American about their political beliefs, you could make much better predictions than you can with real polls, which have to take random samples and thus accept some level of stochastic polling error. On the other hand, polls can also suffer from methodological error, which is fundamentally different in nature; it can be highly pernicious, but does require a smaller quantity of information to correct for. And so on.
Re polls: still more complicated than that, even with 100% polling you'd still have response error and non-voters. Much of the difference between polls is their assumptions about demographics of voters.
And yes, my discussion of polling error was oversimplified, but I was mainly just trying to explain what I meant about having to treat something as random if it’s theoretically predictable given enough information.
EDIT: Hold on; I don’t think 538 does include error bars for their probabilities. They have error bars for measurables like the vote share, but that’s normal. So I don’t understand what this argument is even about :)
If you’re buying and selling bets on a range of events, you can only be immune from arbitrage if all your bets are mutually consistent. For example, if you’re offering 3:1 odds on event X, you can’t offer more than 1:3 on the complementary event ~X (where exactly one of X and ~X must happen). That’s pretty obvious, but there a lot of much trickier variations.
“Nate Silver has a probability model” just means his forecasts are derived from a self-consistent model about the probabilities of all the micro-events that make up the election. That fits with the information FiveThirtyEight have published about their methods, so it seems correct to me.
One way Silver’s model could be invalid is if there’s just a bug, and a single event is tracked in multiple inconsistent ways. For example, if the model unknowingly projected odds for the same politician in both the presidential election and a senate seat, without accounting for the fact that you can’t win both at once. But if that’s the criticism Taleb is making, he should be able to put his finger on a specific bug, which he doesn’t seem to have done.
Another way Silver could be wrong is if his models are far too volatile, so you know if the probabilities shoot up or down a long way they’ll almost certainly revert to the mean again. I imagine you could run arbitrage against that.
It’s far from clear that that’s the case, though. It could be that his models are volatile but still pretty close to reality, in which case just betting against every big swing in Silver’s model would be like trying to make money by predicting day-today movement on the stock market.
That sounds like it might be right. I think another commenter here correctly identified that it’s a misunderstanding -- Silver’s model estimates “the election result of the election were to happen tomorrow”, so unlikely events in the coming months don’t need to be accounted for by definition.
If so, I think there’s a valid criticism that the “now-cast” notion is a bit deceptive, and also impossible to validate (we’ll never know what that hypothetical election result would actually have been).
The nowcast does, but the nowcast isn't the main output, the two (polls-plus and polls-only) forecast models are.
I'll do the author the courtesy of assuming they've got a more expanded version of their argument in their head and just failed to express that, but I don't think that follows at all. I don't think there's a constraint on "Nate Silver has a probability model" that mathematically guarantees "Nate Silver's probability model" is correct, in any sense of that term. One could certainly provide evidence and/or proofs that Nate Silver's model follows those rules, but I don't think it just goes without saying. If nothing else the model could contain an straight-up error; goodness knows that's easy enough.
(If one wishes to add to the concept "probability model" that it must be correct by definition, than I can't accept the statement "Nate Silver has a probability model" with mathematical levels of confidence. He claims to have one, yes, but that's a long way from a proof.)