News for hackers. Stuff that matters.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @06:23AM (#11985221)
Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high probability is a circular definition at best.

Does Bayesian thinking just deny the existence of intrinsic probabilities or considers them out of scope? Assuming there is such a thing as true randomness, for example in quantum physics (and it is not due to our ignorance of hidden variables like in Bohemian mechanics), could a Bayesian assign probabilities to the outcomes? Would a Bayesian be mislead into assigning something other than 50/50 for spin up and spin down if he, by chance, only observes spin up although he is really confident that this is just a statistical fluke? If he is mislead, does that mean that uncertainty about the truth of his assumptions creeps in in case the system momentarily deviates from the expected behavior?

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by jordigh on Monday June 27, 2016 @07:48AM (#11985544)
> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means?

Yeah, Kolmogorov's axioms:

https://en.wikipedia.org/wiki/Probability_axioms

To interpret these axioms for 50% probability means that the measure underneath the density function corresponding to the event "heads" is one-half.

But "rigourous" doesn't have anything to do with the natural world. You can't make physics "rigourous", for example, but you can make the mathematics inspired by physics rigourous. Kolmogorov's axioms just give a mathematical description of probability in a formal sense i.e. only discussing its form, not its meaning. Formalism is about saying, "whatever this means, this is how it should behave". It's a very 20th century notion of mathematics.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @09:08AM (#11985890)
I know the Kolmogorov's axioms but I am really more interested in that part they avoid - what is the meaning of a probability of 0.5? It is surly nice that we can operate with probabilities in a (hopefully) self-consistent way, but it bugs me quite a bit that I don't really precisely unterstand what the result of a calculation implies for the real world.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by jordigh on Monday June 27, 2016 @09:18AM (#11985949)
The modern approach to mathematics is that there is no "meaning", just like "2" or "derivative" has no meaning. We just say how it behaves, or define it in terms of other things, which eventually bottoms out with undefined terms, such as sets and set membership. This is formalism.

How you apply mathematics to the world is not the business of formal mathematics. Whatever you want to do with it is "mere" philosophy. ;-)

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @09:26AM (#11985995)
That is what I meant - I am more asking for a solid philosophical interpretation than a mathematical theory, after all frequentism, Bayesianism and all the other interpretations seem to carry quite a bit of philosophy.

On the other hand in another comment it just boiled down to the question whether there is a measure that gives 1 for the set of all infinite binary sequences with 50/50 zeros and ones and 0 for the set of all the other sequences. So it is not pure philosophy what I am interested in.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by rer0tsaz on Monday June 27, 2016 @07:46AM (#11985533)
No, but let me quote Jaynes (Probability Theory: the Logic of Science, 10.10):

> ‘When I toss a coin, the probability for heads is one-half.’ [...] the issue is between the following two interpretations:

> (A) ‘The available information gives me no reason to expect heads rather than tails, or vice versa – I am completely unable to predict which it will be.’

> (B) ‘If I toss the coin a very large number of times, in the long run heads will occur about half the time – in other words, the frequency of heads will approach 1/2.’

These are not the same except for special circumstances like controlled experiments. Frequentism usually assumes or restricts itself to those special circumstances. The long run here means `for any ε > 0, the probability that the observed frequency n/N lies in the interval (1/2±ε) goes to 1 as N goes to infinity'.

There is no such thing as an intrinsic probability, a coin only has a chance of landing heads when tossed. If we know everything about the coin and how it is tossed we could calculate the result. Some ways of tossing a fair coin are biased. Some coins are biased when tossed in fair ways. A fair coin toss exaggerates factors that are difficult to know and control exactly, like the force that we apply on the coin with our finger.

Jaynes thinks that `true randomness' such as is postulated by conventional quantum physics is unscientific (10.7). In any case it doesn't matter for calculating, and I don't think many Bayesians lose sleep over whether it's unknown, unknowable or `true randomness'.

If the Bayesian has an informative prior, they won't be mislead too much. If a Bayesian is 100% sure beforehand with a δ(x - 0.5) prior they won't be mislead at all, but of course no one is ever 100% sure (Cromwell's rule). On the other hand a frequentist might say p<0.05 and be mislead.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by kareemsabri on Monday June 27, 2016 @08:09AM (#11985627)
> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair.

It doesn't really require a theory, outside of probability theory itself.

According to the law of large numbers, P(x) = 0.5 does imply that, for N trials, the probability of N/2 outcomes being x (heads) approaches 1 as N approaches infinity. And as you pointed out, the probability of all heads is vanishing, as N approaches infinity. P(all heads) = 1 - P(not all heads), and P(all heads) approaches 0 as N approaches infinity, so P(not all heads) must approach 1. The same exercise could be done with the relative distributions of heads and tails, in a large number of trials.

Each coin flip is random, but the results will converge to the mean. You will obtain the same number of heads and tails with certainty, not high probability, if you can make N big enough.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @08:31AM (#11985721)
That means it is totally impossible to get H H H H... or H T T H T T... ad infinitum? I really have no well-founded opinion on that, it just seems very counterintuitive that the coin is not allowed to yield any such sequence, they same as good as any other sequence to me.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by kareemsabri on Monday June 27, 2016 @08:44AM (#11985773)
Not at all! It yields that sequence all the time, in small numbers. But, it is impossible to get only heads, if you could truly flip a coin infinitely many times, which of course you can't. The larger you make the sequence, the smaller you make the probability of all heads. In your example:

P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625

As you can see it's getting pretty small already. However, the probably will never actually reach 0. It's the half step concept, if you keep halving the distance between yourself and an object, you will never actually reach it. But the distance between yourself and the object will approach 0 as the number of steps approaches infinity. 0.5 ^ infinity = 0

Note that P(H|H,H,H) = probability of heads, given 3 previous heads, is still just 0.5. Any independent coin flip is 0.5 chance of being heads. But, when you've done 0 coin flips, and you ask, what are the odds of all heads for the next, say, million flips, they are nearly 0.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @08:59AM (#11985842)
But now you are contradicting yourself, aren't you? We agree that for any finite sequence of tosses it is unlikely but possible to get all heads and therefore it is possible to not converge to 0.5. The question is what difference it makes to go from a large but finite to an infinite number of tosses. Either it is impossible to get only heads an infinite number of times, then I have a problem understanding why that is, or all heads is still a possible outcome even in the infinite case, then the process does not necessarily yield 0.5 even in the limit of an infinite number of tosses.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by kareemsabri on Monday June 27, 2016 @09:07AM (#11985885)
> The question is what difference it makes going from a large but finite to an infinite number of tosses.

Convergence is only guaranteed as N -> ∞. The difference between large but finite and infinite is.. well, infinite :) So that's a pretty significant difference.

> Either is impossible to get only heads an infinite number of times, then I have a problem understanding why that is, or all heads is still a possible outcome even in the infinite case, then the process does not necessarily yield 0.5 even in the limit of an infinite number of tosses.

It's the former. It is impossible to get only heads an infinite number of times. It is possible, but increasingly unlikely, to get only heads a REALLY LARGE, but finite, number of times.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @09:20AM (#11985961)
Okay, assuming that is true, is there an intuitive way to understand that? And the fact that lim[n-> ∞] 0.5^n = 0 unfortunately won't do for me, that is true for any specific infinite sequence, even those containing 50/50 heads and tails. I think most of the sequences - hand-waving, most of an infinite set - are 50/50 heads and tails just because there are more possibilities - hand-waving again - to arrange 50/50 heads and tails versus 100/0 or 40/60 heads and tails. So can I sum over all the 50/50 sequences and get 1 and sum over the rest and get 0? I still would not really understand what forces my coin to show tails eventually, but if there were a measure showing that those two sets have measure 1 respectively 0 it would already be easier to swallow.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by jerf on Monday June 27, 2016 @09:59AM (#11986168)
"Okay, assuming that is true, is there an intuitive way to understand that?"

I mean this quite seriously and am not being sarcastic or dismissive: Probably not. I think the general concept of "intuitive" is that there is some experiential analog to the concept in the real world, with which we've had a lot of experience and can thus "intuit" the behavior. The real world does not include infinity.

You can develop mathematical intuitions, but I don't think that's what you were saying.

In the mathematical intuition sense, it's worth pointing out the probability of any given infinite series of coin flips is zero. The all-heads or all-tails series are not special that way. While this may not be the best way to intuit it, if a metaphysical you who would live forever sat down and started flipping coins, you will never at any point be done with flipping an infinite series of coins. You could sit there until you flip any finite series of coins, but you will never flip an infinite one, even with our unrealistic stipulations of life span. From that perspective, a probability of zero of flipping an infinite number of heads should seem reasonable; there is probability zero that you will ever have flipped an infinite number of coins.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by kareemsabri on Monday June 27, 2016 @09:59AM (#11986166)
> Okay, assuming that is true, is there an intuitive way to understand that? And the fact that lim[n-> ∞] 0.5^n = 0 unfortunately won't do for me, that is true for any specific infinite sequence, even those containing 50/50 heads and tails.

That's true, but only if you pick ONE specific sequence. The likelihood of any precise sequence of heads/tails is equally unlikely as all heads. However, we are not fixing the outcomes ordering, but simply saying taken to infinity the distribution of outcomes will tend to 50/50.

> I still would not really understand what forces my coin to show tails eventually

Nothing forces it to show heads either.

> but if there were a measure showing that those two sets have measure 1 respectively 0 it would already be easier to swallow

Well, we've already proven it's impossible to have all heads infinitely. The same would be true of any non-uniform sequence. Any sequence that favours heads by any margin at all, taken to infinity, would necessarily contain an infinite sub-sequence that contains all heads, which we've proven is impossible by the limit to infinity.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @05:17PM (#11989578)
Well, we've already proven it's impossible to have all heads infinitely. The same would be true of any non-uniform sequence. Any sequence that favours heads by any margin at all, taken to infinity, would necessarily contain an infinite sub-sequence that contains all heads, which we've proven is impossible by the limit to infinity.

Have we? We only said that the probability of having an infinite sequence of all heads is zero but not that it is impossible to obtain that result. After all the probability of any infinite sequence is zero and therefore if we would equate probability zero with impossible to obtain, then we could obtain no infinite sequence at all. The second part is obviously wrong, H H T H H T... ad infinitum obviously favors heads over tails but does not contain an infinite subsequence of all heads.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by conjectures on Monday June 27, 2016 @01:10PM (#11987593)
When heads has probability p the probability for any n-length sequence of all heads is p^n. For p between 0 and 1 this vanishes as n goes to infinity.

The reason is that the conjunction of some events is always less (than or equal to) probable than the events themselves. This should make intuitive sense as the middle bit of a Venn diagram. Even more so for a conjunction of events with a conjunction of events. When taken to the infinite limit you tend to end up with the conjunction having probability 0 or 1.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @05:22PM (#11989602)
I have no problem with that, all infinite sequences of coin tosses have probability zero. The question is whether a infinite sequence of all heads is an obtainable outcome which would be an counterexample to the statement that the relative frequency converges to 0.5 in the limit.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by paol on Monday June 27, 2016 @06:50AM (#11985317)
> doesn't mean that in the long run you will obtain the same number of heads and tails

IANAM, but in my understanding that's exactly what it means, if you express "in the long run" as "the ratio tends to 0.5 as the number of tosses tends toward ∞"

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @07:15AM (#11985412)
IANAM either, but that seems to be wrong to me. For any finite number of tosses you may certainly not converge to 0.5 even if it becomes very unlikely pretty quickly not to do so. In order for this to be true, it would have to be literally impossible to always get heads on every toss no matter how often you try. But it seems counter-intuitive at the very least that it is inevitable to get tails eventually. What rules out the possibility to get heads over and over again, an infinite number of times? [1] But if it is possible to obtain an infinite number of heads and not a single tails, then it is not true that ratio always converges towards 0.5. In consequence one would have to modify the definition and say that it is only very likely to converge towards 0.5 but that obviously raises the problem that »very likely« sounds a lot like »with high probability« which is the thing we are trying to define, hence this would become a somewhat circular definition.

[1] Actually it must not only be impossible to only ever get heads or tails, it must be impossible to obtain any sequence of outcomes that has not exactly the same number of heads and tails. There is only one sequence of outcomes with only heads and only one with only tails but there is already an infinite number of sequences with all heads but one tails or vice versa. And there is an enormous number of possible sequences with 40 % heads and 60 % tails or any other ratio other than 50/50.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by daemonomania on Monday June 27, 2016 @07:33AM (#11985497)
If a coin were flipped arbitrarily many times, and it landed on heads each time, and if we had no other information about the coin, a frequentist would say that the probability the coin lands on heads is one.

Now, you might protest and say, "But this is a fair coin that just happened to land on heads arbitrarily many times". But then you are relying on a prior definition of probability in order to justify your objection to the frequentist's claim, since presumably what you mean by "fair" is that the probability the coin lands on heads is 1/2.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @08:02AM (#11985592)
I am not sure if that really addresses my issue. My problem is that I don't see what eventually forces convergences to 50/50. We can use two or even better many coins or, at least superficially equivalent, one coin and let several experimenters take turns. The results will converge towards 50/50 but only with high probability, or at least I don't see why they alway would, i.e. why not at least one sequence could not converge indefinitely. And that seems to lead to an infinite tower of experiments.

When tossing a coin infinitely often the ratio of heads and tails will converge to 50/50 but only in that sense that you have to toss the coin an infinite number of times an infinite number of times. And you can't really stop here because you may still deviate arbitrarily far from the expected outcome. So you toss the coin an infinite number of times an infinite number of times an infinite number of times. And you have to keep nesting your experiments until you made an infinite tower of experiments.

And I have no idea if that would be sufficient to ensure that you can no longer deviate from the expected outcome, my intuition tends towards no, but then again this is certainly far outside of the realm of things for which I would intuition expect to work. Likely I am really just confused and missing something obvious.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by mvf0 on Monday June 27, 2016 @10:40AM (#11986415)
Your problem is that you don't seem to have sufficient understanding of the concept of limits.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by paol on Monday June 27, 2016 @07:23AM (#11985439)
> For any finite number of tosses you can certainly not converge to 0.5

I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem.

[0] https://en.wikipedia.org/wiki/Limit_of_a_sequence

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @07:37AM (#11985513)
Thanks, fixed. It understand that it is the limit, but the question I am asking is whether it is possible to always get heads, an infinite number of times. If this is possible, then it is not true that the relative frequency always approaches 0.5 as the number of repetitions goes to infinity. I understand that the probability of obtaining heads an infinite number of times is zero, but does that mean it is impossible? After all any specific infinite sequence of outcomes has probability zero.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by paol on Monday June 27, 2016 @08:26AM (#11985702)
> the question I am asking is whether it is possible to always get heads, an infinite number of times

No. By my definition (lay, may be wrong), the probability is the ratio of heads to tails to which we converge at the limit (infinity). The only way for it to converge to 0 is if the probability of heads was 0 to begin with, which would contradict the initial assertion that it was a fair coin.

By the way, from your previous comment it seems to me that you're struggling with getting an intuitive grasp on arguments that rely on infinity. To most people (me included for sure) it's initially difficult to accept that infinity is qualitatively different than a finite amount. Our brain seems to superficially accept the concept, but really it keeps trying to reason about it as a really large (but still finite) quantity.

Are you familiar with the argument of whether 0.999... equals 1? (Spoiler: it does) Some people have a really hard time coming to terms with it, and the fundamental problem for many comes down to the same difficulty of reasoning about infinity. Also, it's a fun way to troll people :)

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @06:02PM (#11989884)
No. By my definition (lay, may be wrong), the probability is the ratio of heads to tails to which we converge at the limit (infinity). The only way for it to converge to 0 is if the probability of heads was 0 to begin with, which would contradict the initial assertion that it was a fair coin.

This seems problematic to me. You can not perform an infinite number of coin tosses and know to what the relative frequency converges. So how do you conclude that a fair coin has probability of 0.5 for heads then? I can't really put my finger on it, but that argument is somehow circular. The probability is what the relative frequency converges to and I can not get anything other than 0.5 because that means to probability would have to have been not 0.5 to begin with. I can really only say that I disagree, I just can't pin it down exactly.

Are you familiar with the argument of whether 0.999... equals 1? (Spoiler: it does) Some people have a really hard time coming to terms with it, and the fundamental problem for many comes down to the same difficulty of reasoning about infinity.

I am aware of that and it seems totally obvious to me. And I also certainly know that infinity is not just a really huge number, but I am not sure I internalized that well enough to not make any mistakes because of the difference. Actually I am pretty sure I make mistakes because of that.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by kgwgk on Monday June 27, 2016 @03:18PM (#11988692)
There are different ways to "approach 0.5 as the number of repetitions goes to infinity": https://en.wikipedia.org/wiki/Convergence_of_random_variable...

I think in this case you have "convergence in probability" (but I've not read carefully the discussion). There is a stronger form, "almost sure convergence", where there is exact convergence with probability one (i.e. in some cases there is no convergence, but those happen with probability zero).

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @05:43PM (#11989754)
That is exactly what I mean. We try to establish what a probability of 0.5 means, i.e. that the relative frequency converges to 0.5 if the number of trails goes to infinity, but that is not exactly true because there is a (possibly vanishingly small) set of outcomes where the relative frequency does not converge to 0.5. This in turn forces us to state that the relative frequency only converges with high probability, but now we have used a probability in our definition of probability, i.e. we made some kind of circular argument. I just skimmed the article you linked and I am not sure if I read it before so I will read it again, but from a first glance it does not look like any definition in there is suitable to avoid the problem.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by kgwgk on Tuesday June 28, 2016 @01:57AM (#11991754)
There is no circularity. You're discussing the real-world interpretation of the probability of an event in terms of long-term frequencies. The probability in the definition of convergence is a well-defined mathematical concept that has nothing to do with frequencies.

Edit: thinking more about it, I agree that the passage from mathematical probability to real-world probability is not very satisfactory. But it's not really surprising, the very notion of "real-world" is troublesome.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by effie on Monday June 27, 2016 @04:49PM (#11989351)
"whether it is possible to always get heads, an infinite number of times" No if we actually make the experiment, for we cannot toss coin infinitely many times. Yes, if we are just considering and measuring all possible infinite sequences. Measure of such mathematically possible sequence is zero, like measure of a point somewhere in a circle is zero. The point exists and is a possible result of choosing a point, but its probability is zero. Thus zero probability does not mean thing is impossible; it is only very unlikely.
Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by danbruc on Monday June 27, 2016 @05:33PM (#11989683)
If an infinite sequence of all heads is a possible outcome, then it is not true that the relative frequency converges to 0.5 in the limit. It is obviously true for many infinite sequences but if all heads is a possibility then it is not true in general.

Just to clarify, I am talking about the frequentist's claim that in the long run the relative frequency becomes the probability, i.e. P(x) = lim[n -> ∞] nx / n where n is the number of trails and nx is the number of trails that yielded x. So again, if an infinite sequence of heads is a possible outcome, then that would mean P(heads) = 1 while the frequentist asserts that it necessarily must become 0.5.

Re:How Bayes’ Rule Emerged Triumphant from Two Centuries of ... by effie on Wednesday June 29, 2016 @01:43PM (#12003239)
Does the frequentist view really claim that infinite sequence nx/n always converges, for all possible runs? That is obviously false. The laws of large numbers only state that probability that the limit of nx/n is equal to 1/2 is 1. This is because all sequences are equally probable and there are overwhelmingly many more sequences that limit to 1/2 than sequences that do not. That is different from saying that the limit is always equal to 1/2, because there are these exceptional cases of low measure like HHHHH..., HTTHTTHTT...