People definitely study whether AGI is possible. For example, philpapers.org lists ~2000 on the philosophy of cognitive science [1], and each of these papers is discussing some aspect of the possibility of AGI.

[1] https://philpapers.org/browse/cognitive-sciences

Every such paper I've seen assumes AGI is possible, and is only concerned with the how question. Alternatively, it talks about 'qualia' and 'consciousness' which are not well defined enough to answer the question whether AGI is possible.

If you've seen any papers addressing the more fundamental question in a quantitative manner I'd be interested to see a link.

> Every such paper I've seen assumes AGI is possible, and is only concerned with the how question.

Human brains exist, human brains are AGI, human brains are manufactured on a daily basis, ergo, it is in theory possible to manufacture an AGI.

How do you know human brains are AGI?

Define AGI.

Reducible to a Turing machine.

That doesn't sound like a complete definition of the term "AGI", unless you mean that anything reducible to a Turing machine is an AGI.

In any case, I'm not sure why the term "artificial general intelligence" should be constrained to Turing machines. Nothing about it implies a Turing machine; just that it possesses intelligence and is artificially manufactured.

Roger Penrose is a pretty famous mathematician who has a side-hobby of arguing that AGI is impossible. I believe his most famous work in this vein is https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind, but he has a handful of others as well. His arguments are pretty well grounded in both math and physics. I wouldn't describe the work as "quantitative", but it's probably about as quantitative as is possible at this point.

> Roger Penrose is a pretty famous mathematician who has a side-hobby of arguing that AGI is impossible.

If AGI is impossible, natural intelligent is either impossible or non-physical, but Penrose’s argument seems be that computing is insufficient for AGI, not that AGI is impossible.

Even then, much of his argument is that there must be something "quantum" happening in the human brain, but he apparently fails to notice that a transistor is a quantum device or otherwise articulate why a mechanical object cannot possibly do all the quantum things a neuron can presumably do. Or why a computer model of that process would never be sufficient to emulate it. Half his arguments are misrepresentations of other people's positions and vague pseudo-mathematical conclusions masquerading as indisputable fact.

All-in-all, that book is a big nothing.

I agree with your synopsis. Penrose raises interesting questions, but his answers are not very convincing. But then again, no materialist answer to his question is very convincing. Which gives us two options:

1. ignore the question by labeling it nonsensical

2. expand our range of hypotheses to include immaterial answers

(2) has never provided a convincing answer to any question, and there's no reason to believe it will here either.

Sure it has. Modern physics depends on a number of concepts that were originally considered impossible for materialist explanations, such as any sort of "pulling" force, field effects, wave particle duality, and more recently with the refutation of local realism. Originally, materialists thought everything had to reduce to bumping billiard balls.

Math itself is necessarily immaterial, and modern science would completely collapse without mathematics.

So, it seems prima facie your claim that immaterial explanations have never been able to adequately answer any question to be incorrect. I see no problem with also proposing an immaterial soul as a scientific hypothesis, if we are able to make the claim quantifiable and empirically testable.

> Modern physics depends on a number of concepts that were originally considered impossible for materialist explanations,

It depends on a lot of things that were considered impossible, but not things that are logically incompatible with materialism.

> So, it seems prima facie your claim that immaterial explanations have never been able to adequately answer any question to be incorrect.

No, all the things you point to are material explanations.

Well, that gets to the question of how we define materialism. Broadly enough, then a nonphysical soul can also be a new materialistic cause, so we do not have a disagreement.

The fundamental issue is whether the new theory, materialistic or not, can be quantitatively and empirically tested in some way. I propose the notion of the mind as a halting oracle is such a thing. And we can call it a materialistic halting oracle so we don't violate the need for purely materialistic explanations.

https://am-nat.org/site/halting-oracles-as-intelligent-agent...

> Math itself is necessarily immaterial, and ...

No, that strikes me as false. Geometry is very material and can explain a good bunch of mathematics. Differential geometry, an important aspect of modern AI, is necessarily symbolic, but that does not exclude its materialistic applications.

If math is material, then we can destroy it. So, for instance, I could destroy the number 1. That doesn't make any sense. Even if the entire universe were destroyed, mathematics would still exist. Recreate the universe again and math would remain unchanged.

Planet youhaventseenit in the Horse Nebula is material. Can you destroy it? No! Could it be destroyed by an unfortunate event? Perhaps. But if the number 1 is just an abstract thought, then that thought can seize to exist.

> Even if the entire universe were destroyed, mathematics would still exist.

You presuppose that maths is immaterial to proof that it's immaterial. That stance cannot be argued with, because you cannot show it to be true. If thought seized to exist, there would be nothing left to argue with, so this is reductio ad absurdum either way leading to a false dychotomy. What remains is: "maths" does exist. So my stance holds as much explanatory power as yours, no more, no less, but yours is metaphysical. You idealize things you don't understand, unless you have a concrete plan of action to "Recreate the universe again".

In philosophy, one opinion is against the notion of the more fundamental question. I see that as institutional philosophy handing off its responsibility, quite paradoxically, because the argument is directed against basal sciences (bio, chem etc.) which give phil a run for its money.

Considering the commercial benefits of non-AGI machine learning, the money is not really thrown away. And many of the potential negative effects of AI can occur even without AGI, so those are worth researching too.

I'd be curious to see just how large the commercial benefits of standard ML actually are. The only reason it is hyped right now is because the media is leading people to believe something close to AGI is right around the corner, because we can bruteforce Go and index a million image dataset...

Anyways, all the AI/ML hype is generated not by actual commercial value, but implied AGI. So, it would behoove us to question the underlying assumption that AGI is actually possible. After all, it is the scientific thing to do.

Machine learning is used widely across a huge number of industries and fields from internet search, pharmaceuticals, mining/energy, digital security, entertainment, etc... So the commercial benefits are definitely tangible and not just "media hype".

On computability of intelligence. I'm not an expert on this, but many people study the dynamics of biological neural networks and can represent these dynamics as PDEs which can then be mapped to electrical circuits. Granted approximations happen along the way and it has been difficult to scale these methods to large population of neurons. It still points to a solid argument that biological neural networks can be represented on a silicon substrate. This is basically what the entire field of neuromorphic engineering is focused on.

My understanding is the practical benefit of machine learning and control systems is mostly due to simple models. Not the fancy "deep" models currently in vogue. An added problem is the high dimensional models are essentially black boxes, and are probably significantly overfitting the data, hence all the adversarial type research.

Why does the mapping of biological neural networks to silicon substrate imply the human mind is a computer?

> Why does the mapping of biological neural networks to silicon substrate imply the human mind is a computer?

Is this a rhetorical question or something, because it seems to me you've answered yourself there. I mean, if the mapping works, what else should it imply besides the consequent?

The implication requires a further premise that the mind is reducible to the brain, which we do not know to be true.

Not just the brain, but the whole body and, even more generally, to phenomena that can be described by physics. Unless you are trying to argue for a non-physical (i.e. magical) soul, the argument is sound.

Right, why assume the mind reduces to physics? This is usually how people argue for AGI being inevitable, but assuming the mind reduces to physics is a big assumption. Perhaps we have a physical soul.

Why not? Everything so far had been reduced to physics; sometimes to at-the-time undiscovered physics.

Deep models on GPUs have made using machine learning on images tractable. Manufacturing is one industry which is expected to benefit a lot from this, using computer vision heavily for automation, quality control and robotics.

I'd be curious to see the actual ROI on this claim.

Well, humans are an AGI and they are possible, therefore AGI is possible.