Yeah, Kolmogorov's axioms:
https://en.wikipedia.org/wiki/Probability_axioms
To interpret these axioms for 50% probability means that the measure underneath the density function corresponding to the event "heads" is one-half.
But "rigourous" doesn't have anything to do with the natural world. You can't make physics "rigourous", for example, but you can make the mathematics inspired by physics rigourous. Kolmogorov's axioms just give a mathematical description of probability in a formal sense i.e. only discussing its form, not its meaning. Formalism is about saying, "whatever this means, this is how it should behave". It's a very 20th century notion of mathematics.
How you apply mathematics to the world is not the business of formal mathematics. Whatever you want to do with it is "mere" philosophy. ;-)
On the other hand in another comment it just boiled down to the question whether there is a measure that gives 1 for the set of all infinite binary sequences with 50/50 zeros and ones and 0 for the set of all the other sequences. So it is not pure philosophy what I am interested in.
> ‘When I toss a coin, the probability for heads is one-half.’ [...] the issue is between the following two interpretations:
> (A) ‘The available information gives me no reason to expect heads rather than tails, or vice versa – I am completely unable to predict which it will be.’
> (B) ‘If I toss the coin a very large number of times, in the long run heads will occur about half the time – in other words, the frequency of heads will approach 1/2.’
These are not the same except for special circumstances like controlled experiments. Frequentism usually assumes or restricts itself to those special circumstances. The long run here means `for any ε > 0, the probability that the observed frequency n/N lies in the interval (1/2±ε) goes to 1 as N goes to infinity'.
There is no such thing as an intrinsic probability, a coin only has a chance of landing heads when tossed. If we know everything about the coin and how it is tossed we could calculate the result. Some ways of tossing a fair coin are biased. Some coins are biased when tossed in fair ways. A fair coin toss exaggerates factors that are difficult to know and control exactly, like the force that we apply on the coin with our finger.
Jaynes thinks that `true randomness' such as is postulated by conventional quantum physics is unscientific (10.7). In any case it doesn't matter for calculating, and I don't think many Bayesians lose sleep over whether it's unknown, unknowable or `true randomness'.
If the Bayesian has an informative prior, they won't be mislead too much. If a Bayesian is 100% sure beforehand with a δ(x - 0.5) prior they won't be mislead at all, but of course no one is ever 100% sure (Cromwell's rule). On the other hand a frequentist might say p<0.05 and be mislead.
It doesn't really require a theory, outside of probability theory itself.
According to the law of large numbers, P(x) = 0.5 does imply that, for N trials, the probability of N/2 outcomes being x (heads) approaches 1 as N approaches infinity. And as you pointed out, the probability of all heads is vanishing, as N approaches infinity. P(all heads) = 1 - P(not all heads), and P(all heads) approaches 0 as N approaches infinity, so P(not all heads) must approach 1. The same exercise could be done with the relative distributions of heads and tails, in a large number of trials.
Each coin flip is random, but the results will converge to the mean. You will obtain the same number of heads and tails with certainty, not high probability, if you can make N big enough.
P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625
As you can see it's getting pretty small already. However, the probably will never actually reach 0. It's the half step concept, if you keep halving the distance between yourself and an object, you will never actually reach it. But the distance between yourself and the object will approach 0 as the number of steps approaches infinity. 0.5 ^ infinity = 0
Note that P(H|H,H,H) = probability of heads, given 3 previous heads, is still just 0.5. Any independent coin flip is 0.5 chance of being heads. But, when you've done 0 coin flips, and you ask, what are the odds of all heads for the next, say, million flips, they are nearly 0.
Convergence is only guaranteed as N -> ∞. The difference between large but finite and infinite is.. well, infinite :) So that's a pretty significant difference.
> Either is impossible to get only heads an infinite number of times, then I have a problem understanding why that is, or all heads is still a possible outcome even in the infinite case, then the process does not necessarily yield 0.5 even in the limit of an infinite number of tosses.
It's the former. It is impossible to get only heads an infinite number of times. It is possible, but increasingly unlikely, to get only heads a REALLY LARGE, but finite, number of times.
I mean this quite seriously and am not being sarcastic or dismissive: Probably not. I think the general concept of "intuitive" is that there is some experiential analog to the concept in the real world, with which we've had a lot of experience and can thus "intuit" the behavior. The real world does not include infinity.
You can develop mathematical intuitions, but I don't think that's what you were saying.
In the mathematical intuition sense, it's worth pointing out the probability of any given infinite series of coin flips is zero. The all-heads or all-tails series are not special that way. While this may not be the best way to intuit it, if a metaphysical you who would live forever sat down and started flipping coins, you will never at any point be done with flipping an infinite series of coins. You could sit there until you flip any finite series of coins, but you will never flip an infinite one, even with our unrealistic stipulations of life span. From that perspective, a probability of zero of flipping an infinite number of heads should seem reasonable; there is probability zero that you will ever have flipped an infinite number of coins.
That's true, but only if you pick ONE specific sequence. The likelihood of any precise sequence of heads/tails is equally unlikely as all heads. However, we are not fixing the outcomes ordering, but simply saying taken to infinity the distribution of outcomes will tend to 50/50.
> I still would not really understand what forces my coin to show tails eventually
Nothing forces it to show heads either.
> but if there were a measure showing that those two sets have measure 1 respectively 0 it would already be easier to swallow
Well, we've already proven it's impossible to have all heads infinitely. The same would be true of any non-uniform sequence. Any sequence that favours heads by any margin at all, taken to infinity, would necessarily contain an infinite sub-sequence that contains all heads, which we've proven is impossible by the limit to infinity.
Have we? We only said that the probability of having an infinite sequence of all heads is zero but not that it is impossible to obtain that result. After all the probability of any infinite sequence is zero and therefore if we would equate probability zero with impossible to obtain, then we could obtain no infinite sequence at all. The second part is obviously wrong, H H T H H T... ad infinitum obviously favors heads over tails but does not contain an infinite subsequence of all heads.
The reason is that the conjunction of some events is always less (than or equal to) probable than the events themselves. This should make intuitive sense as the middle bit of a Venn diagram. Even more so for a conjunction of events with a conjunction of events. When taken to the infinite limit you tend to end up with the conjunction having probability 0 or 1.
IANAM, but in my understanding that's exactly what it means, if you express "in the long run" as "the ratio tends to 0.5 as the number of tosses tends toward ∞"
[1] Actually it must not only be impossible to only ever get heads or tails, it must be impossible to obtain any sequence of outcomes that has not exactly the same number of heads and tails. There is only one sequence of outcomes with only heads and only one with only tails but there is already an infinite number of sequences with all heads but one tails or vice versa. And there is an enormous number of possible sequences with 40 % heads and 60 % tails or any other ratio other than 50/50.
Now, you might protest and say, "But this is a fair coin that just happened to land on heads arbitrarily many times". But then you are relying on a prior definition of probability in order to justify your objection to the frequentist's claim, since presumably what you mean by "fair" is that the probability the coin lands on heads is 1/2.
When tossing a coin infinitely often the ratio of heads and tails will converge to 50/50 but only in that sense that you have to toss the coin an infinite number of times an infinite number of times. And you can't really stop here because you may still deviate arbitrarily far from the expected outcome. So you toss the coin an infinite number of times an infinite number of times an infinite number of times. And you have to keep nesting your experiments until you made an infinite tower of experiments.
And I have no idea if that would be sufficient to ensure that you can no longer deviate from the expected outcome, my intuition tends towards no, but then again this is certainly far outside of the realm of things for which I would intuition expect to work. Likely I am really just confused and missing something obvious.
I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem.
No. By my definition (lay, may be wrong), the probability is the ratio of heads to tails to which we converge at the limit (infinity). The only way for it to converge to 0 is if the probability of heads was 0 to begin with, which would contradict the initial assertion that it was a fair coin.
By the way, from your previous comment it seems to me that you're struggling with getting an intuitive grasp on arguments that rely on infinity. To most people (me included for sure) it's initially difficult to accept that infinity is qualitatively different than a finite amount. Our brain seems to superficially accept the concept, but really it keeps trying to reason about it as a really large (but still finite) quantity.
Are you familiar with the argument of whether 0.999... equals 1? (Spoiler: it does) Some people have a really hard time coming to terms with it, and the fundamental problem for many comes down to the same difficulty of reasoning about infinity. Also, it's a fun way to troll people :)
This seems problematic to me. You can not perform an infinite number of coin tosses and know to what the relative frequency converges. So how do you conclude that a fair coin has probability of 0.5 for heads then? I can't really put my finger on it, but that argument is somehow circular. The probability is what the relative frequency converges to and I can not get anything other than 0.5 because that means to probability would have to have been not 0.5 to begin with. I can really only say that I disagree, I just can't pin it down exactly.
Are you familiar with the argument of whether 0.999... equals 1? (Spoiler: it does) Some people have a really hard time coming to terms with it, and the fundamental problem for many comes down to the same difficulty of reasoning about infinity.
I am aware of that and it seems totally obvious to me. And I also certainly know that infinity is not just a really huge number, but I am not sure I internalized that well enough to not make any mistakes because of the difference. Actually I am pretty sure I make mistakes because of that.
I think in this case you have "convergence in probability" (but I've not read carefully the discussion). There is a stronger form, "almost sure convergence", where there is exact convergence with probability one (i.e. in some cases there is no convergence, but those happen with probability zero).
Edit: thinking more about it, I agree that the passage from mathematical probability to real-world probability is not very satisfactory. But it's not really surprising, the very notion of "real-world" is troublesome.
Just to clarify, I am talking about the frequentist's claim that in the long run the relative frequency becomes the probability, i.e. P(x) = lim[n -> ∞] nx / n where n is the number of trails and nx is the number of trails that yielded x. So again, if an infinite sequence of heads is a possible outcome, then that would mean P(heads) = 1 while the frequentist asserts that it necessarily must become 0.5.
Does Bayesian thinking just deny the existence of intrinsic probabilities or considers them out of scope? Assuming there is such a thing as true randomness, for example in quantum physics (and it is not due to our ignorance of hidden variables like in Bohemian mechanics), could a Bayesian assign probabilities to the outcomes? Would a Bayesian be mislead into assigning something other than 50/50 for spin up and spin down if he, by chance, only observes spin up although he is really confident that this is just a statistical fluke? If he is mislead, does that mean that uncertainty about the truth of his assumptions creeps in in case the system momentarily deviates from the expected behavior?