> A technology so advanced that not only does it subsume all other technology, but it is able to improve itself.

The problem is, a computer has no idea what "improve" means unless a human explains it for every type of problem. And of course a human will have to provide guidelines about how long to think about the problem overall, which avenues to avoid because they aren't relevant to a particular case, etc. In other words, humans will never be able to stray too far from the training process.

We will likely never get to the point where an AGI can continuously improve the quality of its answers for all domains. The best we'll get, I believe, is an AGI that can optimize itself within a few narrow problem domains, which will have limited commercial application. We may make slow progress in more complex domains, but the quality of results--and the ability for the AGI to self-improve--will always level off asymptotically.

Huh? Humans are not anywhere near the limit of physical intelligence, and we have many existence proofs that we (humans) can design systems that are superhuman in various domains. "Scientific R&D" is not something that humans are even particularly well-suited to, from an evolutionary perspective.

> The problem is, a computer has no idea what "improve" means unless a human explains it for every type of problem

Not currently.

I don’t really think AGI is coming anytime soon, but that doesn’t seem like a real reason.

If we ever found a way to formalize what intelligence _is_ we could probably write a program emulating it.

We just don’t even have a good understanding of what being intelligent even means.

> The best we'll get, I believe, is an AGI that can optimize itself within a few narrow problem domains

By definition, that isn’t AGI.

If that is what AGI looks like.

There may well be an upper limit on cognition (we are not really sure what cognition is - even as we do it) and it may be that human minds are close to it.

Yes, we can imagine that there's an upper limit to how smart a single system can be. Even suppose that this limit is pretty close to what humans can achieve.

But: you can still run more of these systems in parallel, and you can still try to increase processing speeds.

Signals in the human brain travel, at best, roughly at the speed of sound. Electronic signals in computers play in the same league as the speed of light.

Human IO is optimised for surviving in the wild. We are really bad at taking in symbolic information (compared to a computer) and our memory is also really bad for that. A computer system that's only as smart as a human but has instant access to all the information of the Internet and to a calculator and to writing and running code, can already be effectively act much smarter than a human.

I think our issue is much more banal: we are very slow talkers and our effective communication bandwidth is measured in bauds. Anything that could bridge this airgap would fucking explode in intelligence.

Yes, that's one aspect.

Our reading speed is not limited by our talking speed, and can be a bit faster.

And that's even more true, if you go beyond words: seeing someone do something can be a lot faster way to learn than just reading about it.

But even there, the IO speed is severely limited, and you can only transmit very specific kinds of information.

Very unlikely, for the reason that human minds evolved under extremely tight energy constraints. AI has no such limitation.

Except also energy constraints.

But I agree, there’s no reason to believe humans are the universal limit on cognitive abilities

The energy constraints for chips are more about heat dissipation. But we can pump a lot more energy through them per unit volume than through the human brain.

Especially if you are willing to pay a lot for active cooling with eg liquid helium.

A constraint is still a constraint

A constraint that's not binding might as well not exist.

Since we do not know what cognition is we are all whistling in the dark.

Energy may be a constraint, it may not. What we do not know is likely to matter more than what we do