If you see it as copying an existing model to another computer, yes it is trivial. But an AGI trying to replicate itself in the real world has to also make those computers.
Making modern computer chips is one of the most non-trivial things that humans do. They require fabs that cost billions, with all sorts of chemicals inside, and extreme requirements on the inside environment. Very hard to build, very easy to disable them via an attack.
Convince a couple rich people that it can cure aging with _just_ enough compute.
Hack datacenters (an AGI would likely be much much much better at finding security holes than humans).
Make _really really funny videos_ and create the most popular Patreon of all time.
If you're 500 IQ or above there are a _lot_ of things you can do, and probably simultaneously, since an AGI probably won't be limited by human's ability to think ~one thought at a time. (Being less than 500 IQ myself I probably haven't thought of everything.)
I imagine an better-than-human AGI could figure this out and plan/act accordingly, considering a mere human being figured this out. (Also sometimes it might be better to think of an AGI, when it gets to a certain level, as a nation state and not an individual)
> trivial
It doesn't need to be trivial to be incredibly dangerous
> The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else
? Why ?
And why does each 'AI' have to map to a single 'computer'?
This is narrow automaton thinking.
'Intelligence' in humans has nothing necessarily to do with real intelligence.
We are automatons with brains with very limited networking capability because of the limits of biology.
And of course what does 'replication' have to do with anything anyhow.
What software can learn on its own without any assistance from a huamn? I've not heard of anything like this.
> Replicate? Trival. But what does that have to do with AGI?
Like humans, an AGI should be able to replicate. Similar to a von neumann probe.
> Solve Problems Humans Cannot. Check.
What unthinkable problem has an AI solved? Is something capable of solving something so grandiose we almost can't even define the problem yet?
Think back to the 1930's when some physicists were mumbling something about "throwing a neutron at an atom that releases two neutrons and release tremendous energy"
First - we already have software that can unequivocally do the things you just highlighted.
Learn? Check.
Replicate? Trival. But what does that have to do with AGI?
Solve Problems Humans Cannot. Check.
So we already have 'AGI' and it's a simple computer program.
Thinking about 'AGI' as a discrete, autonomous system makes no sense.
We will achieve highly intelligent systems with distributed systems decades before we have some 'individual neural net on a chip' that feels human like.
And when we do make it, where do we draw the line on it? Is a 'process' running a specific bit of software an 'AI'?
What if the AI depends on a myriad of micro-services in order to function. And those micro-services are shared?
Where is the 'Unit AI'?
The notion of an autonmous AI, like a unit of software on some specific hardware distinct from other components actually makes little sense.
Emergent AI systems will start to develop out of our current systems long before 'autonomic' AI. In fact, there's no reason at all to even develop 'autonomic AI'. We do it because we want to model it after our own existence.