Re:Making a neural network say “I Don’t Know”: Bayesian NNs ...
by conjectures on Wednesday December 05, 2018 @07:08AM
(#18607545)
> Even in the case of random measures, while you can reasonably talk about the probability of a probability distribution, it still doesn't really make sense to talk about the probability distribution of a probability.
But that's exactly what it does mean. For the Dirichlet Process, the probability of hitting a set in the partition of a sample space is random. A DP is a couple of generalisations up from the beta distribution on p under discussion [1]. I.e. this specializes to the current situation as:
* The sample space theta is the set {H, T}.
* The only interesting partition to talk about is {A_1, A_2}={{H}, {T}}.
* The Dirichlet distribution on (G(A_1), G(A_2)) is therefore 2 dimensional, which is a Beta.
* G(A_1) is the probability measure of hitting heads.
No specific parametric model involving univariate p needs to be involved.
[1] c.f. eqn 2 of http://www.gatsby.ucl.ac.uk/~ywteh/research/npbayes/dp.pdf
So while it is meaningful to say:
"p has this probability distribution."
It is not meaningful to say:
"The probability the coin comes up heads has this probability distribution."
This is because p and the probability the coin comes up heads are not the same thing (this is even more apparent if you have a non-Bernoulli scenario) when p is no longer fixed.
It's not really just a case of marginal vs conditional probabilities either. Even in the case of a conditional probability, it's only defined for a fixed condition.
Even in the case of random measures, while you can reasonably talk about the probability of a probability distribution, it still doesn't really make sense to talk about the probability distribution of a probability.