Even given infinite resources Solomonoff induction cannot be computed, because we computationally cannot eliminate all non halting programs.

Why believe human intelligence is limited by physics?

Because elan vital and phlogiston turned out to be terrible ideas?

Human cognition affects human action which has physical effects. So at the least, intelligence is causally linked with known phycial processes. I'd say that's sufficient to make it amenable to physical inquiry.

Pulling out drastic measures like extraphysical magic, just to hand wave at an imprecise problem seems like an act of epistemic violence or something.

Certainly intelligence is causally linked with the physical world, hence our conversation. But, this does not imply intelligence is itself physical. Additionally, saying intelligence is non physical does not turn it into some kind of inscrutable magic. Could you explain why you think this is the case?

For example, I wrote an article explaining how modeling the human mind as a halting oracle results in empirically meaningful results.

https://am-nat.org/site/halting-oracles-as-intelligent-agent...

I think it depends on how we delineate physical vs non-physical. In this case, I'm taking a simple definition of "physical" to mean "anything amenable to scientific inquiry" or "any phenomenon that is, in principle, part of consensus reality".

Actually, I'm confused what we'd even be claiming by calling intelligence purely non-physical. Perhaps you're thinking of the qualia of intelligence?

Regarding the article you wrote, I think the concept of "partial oracle", while coherent, isn't super helpful. Or rather, it's too broad of a characterization. The halting problem doesn't claim that Turing machines can't solve some halting problems, it's that there's no finite class of turing machines which can together solve any halting problem.

It's not precisely news that some Turing machines can solve the halting problem for certain (infinite) classes of machines. Case in point, regular languages.

Anyway, meta-discussion-wise, I'm super happy to find kindred souls that also enjoy thinking about these things. I recently came across the Complexity Zoo[0] website, and if you're anything like me, I suspect you'll enjoy it too!

[0]:https://complexityzoo.uwaterloo.ca/Complexity_Zoo

I would disagree with that definition of physical. I believe a better definition of physical is anything that can be reduced to the laws of physics. Since the laws of physics are computable, I would enlarge the definition to be anything that can be reduced to a Turing machine. It's logically possible there are irreducible entities, i.e. partial oracles, so it is logically possible there are non physical, yet physically testable entities.

In the article the point is a bit subtle, but partial oracles are still not Turing reducible. The point of the article is that the fact humans cannot solve all problems does not imply they are Turing reducible.

At any rate, from my cursory analysis, it appears the notion of a non Turing reducible human mind is empirically tractable, so I still don't understand why it is not a viable scientific hypothesis and why AGI is assumed to be necessarily true. The mind as a partial oracle would imply AGI is impossible.

Thanks for the link, I'll be checking it out!

How do you know the laws of physics are computable (for a particular sense of "computable")? Perhaps the known physical laws are, but physics is still not a solved problem. In any case, if any new irreducible, physically-interacting elements of reality were to be discovered, there would be no reason not to consider them as irreducible elements of physics.

Yes, as far as we know they are computable. And sure, without a well defined concept of "physical" we can call anything we want physical.

The main point is the brain, as far as we know, is reducible to the known laws of physics, which, as far as we know, are entirely computable. Therefore, if human beings exhibit non computable phenomena, such as functioning as halting oracles, then that is a good reason to believe the mind does not reduce to the brain. And we could call this 'mind' a new form of matter or something, so that people don't get alarmed by non-physical phenomena.

> The main point is the brain, as far as we know, is reducible to the known laws of physics, which, as far as we know, are entirely computable. Therefore, if human beings exhibit non computable phenomena, such as functioning as halting oracles, then that is a good reason to believe the mind does not reduce to the brain.

This is where I disagree. It would, to me, seem to be a good reason to believe we've missed something about the way the body (not specifically the brain) works.

> And we could call this 'mind' a new form of matter or something, so that people don't get alarmed by non-physical phenomena.

This just sounds like you are eager to label any new phenomena "non-physical". There is no reason to pull the extraphysical gun just because you are stumbling upon new physics.

Solomonoff induction can be computed in the limit. During round i, simulate your chosen UTM on each valid input string (note: assuming that the valid input strings are prefix free, which iirc is the standard for solomonoff induction) of length at most i, until you have simulated it for i steps total, or until it halts (whichever comes first).

When doing so, for any string, the proportion of the inputs (with respect to the coin flip measure thing) to the UTM that eventually halt with whatever particular output, which will have been found to halt , will approach "all of them".

Now, you can't quite say "run as long as you need to in order to get answer within this epsilon of correct answer", because you don't have a way of determining whether something hasn't halted yet but will, or if it never will, but, it is still computable in the limit. If you use absurdly (possibly quite absurdly) large but finite amounts of computational resources, it should give you a good enough approximation.

Humans make mental errors often enough. I don't see why one would think that, because solomonoff induction is only approximable, that that suggests human intelligence is uncomputable. Why not suppose that humans just have a very close approximation to the uncomputable thing, rather than the noncomputable thing itself?

Why believe that human intelligence is computable? Well, for me it is more of a "choosing not to believe that it is uncomputable". If I believed it uncomputable, I might hang too much of my philosophy/worldview on that assumption, and then if I turned out to be incorrect on that, too much might come crashing down.

Edit: note: I mean "not believing", not "disbelieving". Not(believe(p)) rather than believe(not(p)) .

Ah, right. That's true. Since the elegant program is guaranteed to halt, then we'll get to it eventually. My mistake. That being said, even with infinite resources we will never know when that actually happens, so we still will never known when Solomonoff induction has succeeded.

I don't see what will come "crashing down." Science is about being able to quantify and empirically test our hypotheses. Nothing about uncomputable intelligence seems to undermine this idea, but perhaps I am missing something. Instead, it broadens the range of hypotheses we can use to explain the world, which seems to be a good thing.

When I said crashing down, I meant in my own beliefs personally, not anything about the impact on science as a whole.

Like, for example, if I built a justification for my believing in the existence of souls on top of the belief that the human mind is uncomputable, then if I turned out to be incorrect about the human mind, it might lead me to believe that souls don't exist (though that wouldn't really be a logical result, but I am not always rational). So, in order to have better foundations for my personal beliefs, I prefer not to rely on the assumption of uncomputability of the mind (nor on its negation).

Basically, I want to avoid being too optimistic and believing too much because of it being convenient for my other beliefs.

If your beliefs are consistent with a computable mind, then what stops you from also investigating the uncomputable mind? If true, then your beliefs are broadened, and if false nothing is threatened.

Ah, I don't mean to reject the possibility that minds are uncomputable. Perhaps they are!

It does seem likely that ideal minds would be (for reasons like those you describe).

I just think I should require a rather strong argument to conclude that they are, instead of remaining non-commital.