> That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?

Yes, with two caveats. First, you can't have assigned zero probability to the right answer. Which means that if you have a probability distribution over hypotheses, and the right answer wasn't one of the hypotheses in the distribution, you're doomed from the start. The Solomonoff prior is a mathematical construct that gets around this problem by including every hypothesis that can be expressed as a Turing machine, but it gets used more as a philosophical token than an actual computing tool because it's unfeasible to use directly. The second issue is that if you have a bad enough prior, "enough additional pieces of evidence" can be an arbitrarily large amount, and there may only be a limited amount of evidence available to collect. In particular, rerunning an experiment over and over can only provide a limited amount of evidence, because of the possibility that some systematic error affects every instance of the experiment.

The prior of Solomonoff Induction is a uniform distribution over the enumerable set of Turing machines?

Ah I dimly remember that it also has an Occams razor built-in, so the prior probability falls off with the lengths of the strings that describe the Turing machines?

In my field (Epidemiology), when doing Bayesian analysis, it is very common to set one's priors to be a distribution. Sometimes the point estimate and spread of a previously conducted study or meta-analysis, sometimes merely a uniform distribution with upper and lower bounds ("It is extremely unlikely that the relative risk of disease for this exposure is below 0.01 or above 100...")

It's been argued that frequentist analysis is essentially a Bayesian analysis with a prior distribution centered on zero with bounds from positive to negative infinity.

How does that work? It doesn't sound like a well-defined distribution since the area under the curve needs to be 1.

It has to do with calculus. If the probability of a given result approaches zero as the result itself approaches positive or negative infinity, then the area under the curve approaches 1.

Imagine 1/2 + 1/4 + 1/8 ... to infinity. The sum approaches 1 as the denominator approaches infinity. With calculus, we can determine that with mathematical methods.

I think he was referring specifically to an unbounded uniform distribution, which is indeed not well-defined.

It's more of a philosophical statement than an actual implementation. Mainly that frequentist analysis begins with the prior "I dunno, could be anything..."

See also: The calculus answer.

It turns out that priors don't actually need to be distributions all the time. This often occurs when seeking maximally "uninformative" priors.

Bayesian analysis comes directly from the probability axioms, which are 'frequentist'.

Huh? You mean the Kolmogorov axioms? In what sense are these 'frequentist'?

The third axiom just counts the event space.

That's the right bottom part of the Bayes' rule?

This comes to mind: https://en.wikipedia.org/wiki/Aumann's_agreement_theorem

Essentially, two genuine Bayesian rationalists (with some hand wavy preconditions) cannot agree to disagree; ie, they will eventually converge onto the same understanding of an event.

This isn't true though, the problem is there are generally more then 2 possible explanations. For instance let's imagine both of us are using an experimental telescope to observe if some event occurs. We are then looking at four possible scenarios. The telescope could work/not work correctly and the event could happen/not happen. You are confident that the telescope works correctly and also confident that the event will not occur so you give high prior probability to the first and a low to the second. I on the other hand think the telescope is rubbish and the event will almost certainly occur and do the opposite. We sit down and wait and do not observe the event. You then come to the conclusion that the event did not occur and the telescope works correctly, while I come to the reverse conclusion.

One of the "handy wavy preconditions" is that they share priors. This nearly never happens in the real world. Almost all disagreements can be traced to differing priors.

No - the precondition is that agents share common knowledge of each others' priors (not that they have the same priors).

Here's the paper I read: http://www.ma.huji.ac.il/~raumann/pdf/Agreeing%20to%20Disagr...

Here's how it starts: "Theorem: if two people have the same priors..."

Whoops, seems you are correct. I'd misunderstood this, my apologies.

First Qn: yes. But the rabbit is hiding in the 'enough'. Most commonly the poor uses of BT end up with a narratively argued prior + a single suite of evidence.

For a finite set of evidence (particularly chosen by someone with bias), bias + evidence can be arbitrarily far from the truth.

> Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?

Almost everyone does this, and the solution is a posterior probability distribution. Most uses of Bayesian techniques that I'm familiar with are based on Monte Carlo simulations, where priors are drawn from the specified prior distribution, processed in some fashion, and result in samples of the posterior distribution.

This is especially helpful when you don't view the problem as having a 'true' single answer with some 'uncertainty' but instead have actual variability in the system, which is described by the posterior distribution.

Thinking about it, I'm really not sure in what instance the prior wouldn't be a distribution. It is, by definition, a probability. I suppose you could have a singular value, where the pdf is a delta function, but what's the point of doing the Bayesian inversion or estimation then? If you have a single value with prior p(x) = 0 or 1, you should end up with the posterior p(x|d)=0 or 1.

So the simplest case is where the prior is a binary: Say p(x=yes) = 0.6, p(x=no)=0.4. Or something like that. It's not a continuous distribution but it's still a distribution. The sum/integral of all cases has to be 1.

Yes, but.

The technical term for the condition you're interested in is consistency. There is a theorem (Doob) regarding the consistency of Bayes estimates. [1]

In simple cases it means that under any non-silly prior your posterior will converge to the truth. E.g. mean of a normal distribution with a normal prior.

The complex cases include things like models that expand when given more data.

[1] http://www4.stat.ncsu.edu/~sghosal/papers/bhurev.pdf

> if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?

That's the general trend: pooling data tends to make priors converge. However, converging priors isn't quite the same thing as everyone converging on the truth.

Statistical expressions themselves are an incomplete explanation, we routinely use assumptions about the direction of causality that aren't captured in them. See Judea Pearl's "Why I am only half Bayesian" paper for a discussion as well as an intro to how his framework approaches such independence assumptions.

Yes and yes and yes!

ETA - caveat to the 2nd question: ... unless you've restricted your prior to exclude the truth.