I'm not at all qualified to talk about this. Without further ado:

Bayesianism has the concept of an A_p distribution, which is roughly "the probability that the probability is p".

My A_p distribution for this coin would be (almost) 50% on 0 and 50% on 1, because I don't think it's possible to bias a coin except by making both faces the same. If I didn't think that, it might be proportional to something like (p-1/2)^2, because more extreme biases seem more likely.

In both cases, my probability of seeing a heads is 1/2, because that's the mean of the A_p distribution. But after flipping the coin once, my A_p distribution updates, and now I can give a probability of seeing the next toss come up heads.

I haven't read the whole of it, but relevant chapter from Jaynes: http://www-biba.inrialpes.fr/Jaynes/cc18i.pdf

There's no need to introduce any special new concepts. It's simply a hierarchical graphical model. You have one variable which stands for p (distributed according to some prior, like a Beta distribution perhaps), and you have another (perhaps multiple), Bernoulli distributed variable whose parameter comes from the first variable and represents the actual coin value that you get.

If you do multiple coin flips, you have more of these Bernoulli variables but they all share a common parameter p.

One such example model is the Beta-Bernoulli process: http://www.math.uah.edu/stat/bernoulli/BetaBernoulli.html

> You have one variable which stands for p (distributed according to some prior, like a Beta distribution perhaps)

This prior sounds like the A_p distribution.

The point is it's a distribution, not a number. You can't just say "I think the probability of seeing heads is p". But if you say "I think the probability of seeing heads is distributed like this, which has a mean of p", then that captures everything you need to know about the coin to update your expectations when you see the results.

(In case it was unclear, A_p isn't a specific shape of distribution, or class of shapes of distributions, like beta or gaussian. It's a distribution over hypothesis-space, where the hypotheses are probabilities, and can have any valid distribution shape.)

I was suggesting that this is less "mysterious" than it seems to sound, once you use a hierarchical model, where one random variable's outcome is used as the parameter for another variable.

Imagine one random variable p at the top center of a page, and then several variables under it, each connected by an arrow coming from p.

The random variables down there are the coin flips (each results in 0 or 1), and they are identically and independently distributed with a Bernoulli(p) distribution. And p itself is a random variable which has a distribution (expressing how likely each p is).

I'm just clarifying that all this fits into normal probability theory, there is no need to introduce any sort of "meta-probability theory".

A biased coin usually means that one side has a greater chance of falling up, but they have both heads and tails, so p(H) + p(T) is still 1

I know that's what it means. I don't think they exist in the real world. See e.g. http://www.stat.columbia.edu/~gelman/research/published/dice... - I don't remember if I've read that specific thing, but I've read things that said the same.

(With skill, it's possible to bias a coin toss, but that's different.)

I was afraid somebody was going to bring up this paper. As the paper shows, they in fact do exist. But only if you spin rather than flip in the normal way.