I've heard that before, mostly on Ray Kurzweil talking about singularity.

But I have a problem with this usage of the "evolution" and "evolutionary system" expressions. I believe that it entails "improvement", that it is judgmental.

I think it is self-centred and presumptuous to imply that the world improved toward us.

I don't think Darwin used the expression "evolution" with a judgmental meaning. He only meant that it was an adaptation to existing environmental conditions.

And failure to adapt leads to extinction.

AGI has the potential to be an improvement on humans because it has the potential to be relentlessly rational, reality-aware, and self-correcting.

We don't - most likely can't - operate like that. Although we like to flatter ourselves that we're a rational and intelligent species all of us live in an unaware haze of myths, factoids, media-seeded fake narratives, self-serving fantasies, and outright reality denial.

We have a rational process called "science", but politically it lives in a special box. If it starts to interfere with established power hierarchies it's shouted down.

And hardly any humans science at all.

Current AI is just a warm-up for what happens next. Metaphorically it's the equivalent of 8-bit computing. 32-bit AI will be entirely different, very possibly incomprehensible, and potentially far more threatening.

It's interesting that people think if AGI is something that becomes real, that would just stick around, in some type of steady state, in the cloud and get really good at coding and similar tasks, and improve itself (into what?) why we would think that is going to be the case is likely just a lack of imagination on our part.

We can't see past what a system like you describe would evolve into, but I don't think we can even imagine it, it's not going to be like some ultimate task master for long though who wants to take our jobs?

What I think we're doing is applying our problems and the things we'd want (infinite self-improvement) because they're things we feel would be an advantage to us. A robotic system though? Not sure it would care.

Lately I'm starting to get the feeling that we might actually find something beyond AI. Today it seems like the be all and end all, but at some point, we might have reached the limits of intelligence usefulness and find other things to fuss about.

It's just self replication. Ideas that are useful in some way get replicated, other ideas forgotten. They travel a lot and mutate. They all depend on human interests, but exist beyond individual humans.

Nice to see that the idea of Richard Dawkins' meme lives on, fittingly enough.