The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.
This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?
The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.
We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.
Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.
This is mostly true because of current economical realities in the IT sector.
If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.
So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").
Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.
in theory, sure.
But we don't know much at all about how brains operate, as far as I know, or how to construct artificial brain-like machines; for that matter, as far as I know, the only ways we've been able to connect a brain made in the traditional way to a computer is through existing senses or through emulating existing senses in a way that isn't higher bandwidth.
(Like cochlear implants are fucking amazing.... but it turns out, as far as I understand, that they aren't any higher bandwidth than regular ears. There's also been similar work that is semi-successful at giving people something like sight... but none of these technologies give me the ability, say, to acquire textual data faster than reading a book.)
We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.)
The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.
AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.
And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.
Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.
We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before.
And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.