> Every such paper I've seen assumes AGI is possible, and is only concerned with the how question.

Human brains exist, human brains are AGI, human brains are manufactured on a daily basis, ergo, it is in theory possible to manufacture an AGI.

How do you know human brains are AGI?

Define AGI.

Reducible to a Turing machine.

That doesn't sound like a complete definition of the term "AGI", unless you mean that anything reducible to a Turing machine is an AGI.

In any case, I'm not sure why the term "artificial general intelligence" should be constrained to Turing machines. Nothing about it implies a Turing machine; just that it possesses intelligence and is artificially manufactured.

Roger Penrose is a pretty famous mathematician who has a side-hobby of arguing that AGI is impossible. I believe his most famous work in this vein is https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind, but he has a handful of others as well. His arguments are pretty well grounded in both math and physics. I wouldn't describe the work as "quantitative", but it's probably about as quantitative as is possible at this point.

> Roger Penrose is a pretty famous mathematician who has a side-hobby of arguing that AGI is impossible.

If AGI is impossible, natural intelligent is either impossible or non-physical, but Penrose’s argument seems be that computing is insufficient for AGI, not that AGI is impossible.

Even then, much of his argument is that there must be something "quantum" happening in the human brain, but he apparently fails to notice that a transistor is a quantum device or otherwise articulate why a mechanical object cannot possibly do all the quantum things a neuron can presumably do. Or why a computer model of that process would never be sufficient to emulate it. Half his arguments are misrepresentations of other people's positions and vague pseudo-mathematical conclusions masquerading as indisputable fact.

All-in-all, that book is a big nothing.

I agree with your synopsis. Penrose raises interesting questions, but his answers are not very convincing. But then again, no materialist answer to his question is very convincing. Which gives us two options:

1. ignore the question by labeling it nonsensical

2. expand our range of hypotheses to include immaterial answers

(2) has never provided a convincing answer to any question, and there's no reason to believe it will here either.

Sure it has. Modern physics depends on a number of concepts that were originally considered impossible for materialist explanations, such as any sort of "pulling" force, field effects, wave particle duality, and more recently with the refutation of local realism. Originally, materialists thought everything had to reduce to bumping billiard balls.

Math itself is necessarily immaterial, and modern science would completely collapse without mathematics.

So, it seems prima facie your claim that immaterial explanations have never been able to adequately answer any question to be incorrect. I see no problem with also proposing an immaterial soul as a scientific hypothesis, if we are able to make the claim quantifiable and empirically testable.

> Modern physics depends on a number of concepts that were originally considered impossible for materialist explanations,

It depends on a lot of things that were considered impossible, but not things that are logically incompatible with materialism.

> So, it seems prima facie your claim that immaterial explanations have never been able to adequately answer any question to be incorrect.

No, all the things you point to are material explanations.

Well, that gets to the question of how we define materialism. Broadly enough, then a nonphysical soul can also be a new materialistic cause, so we do not have a disagreement.

The fundamental issue is whether the new theory, materialistic or not, can be quantitatively and empirically tested in some way. I propose the notion of the mind as a halting oracle is such a thing. And we can call it a materialistic halting oracle so we don't violate the need for purely materialistic explanations.

https://am-nat.org/site/halting-oracles-as-intelligent-agent...

> Math itself is necessarily immaterial, and ...

No, that strikes me as false. Geometry is very material and can explain a good bunch of mathematics. Differential geometry, an important aspect of modern AI, is necessarily symbolic, but that does not exclude its materialistic applications.

If math is material, then we can destroy it. So, for instance, I could destroy the number 1. That doesn't make any sense. Even if the entire universe were destroyed, mathematics would still exist. Recreate the universe again and math would remain unchanged.

Planet youhaventseenit in the Horse Nebula is material. Can you destroy it? No! Could it be destroyed by an unfortunate event? Perhaps. But if the number 1 is just an abstract thought, then that thought can seize to exist.

> Even if the entire universe were destroyed, mathematics would still exist.

You presuppose that maths is immaterial to proof that it's immaterial. That stance cannot be argued with, because you cannot show it to be true. If thought seized to exist, there would be nothing left to argue with, so this is reductio ad absurdum either way leading to a false dychotomy. What remains is: "maths" does exist. So my stance holds as much explanatory power as yours, no more, no less, but yours is metaphysical. You idealize things you don't understand, unless you have a concrete plan of action to "Recreate the universe again".

In philosophy, one opinion is against the notion of the more fundamental question. I see that as institutional philosophy handing off its responsibility, quite paradoxically, because the argument is directed against basal sciences (bio, chem etc.) which give phil a run for its money.