For the relatively small group of researchers and theorists consumed by curiosity about the nature of knowledge, experience, and learning, AGI is a fairly understandable goal.
The more important questions may be how/why/whether entities with emergent-aggregate intelligence (organizations such as corporations, nation-states, political parties, international coalitions, activist movements, schools of thought, etc.) would have AGI as a goal. As individuals, we often tolerate and even willingly engage in (sometimes with zeal) all sorts of pursuits that we're "not on board with" because of the gravity of these aggregates. Even when we realize that the goals of the aggregate are incompatible with our individual goals.
I'm pretty sure that other mammals on this planet are not on board with humans being derived from them, and now humans replacing them and destroying their habitat. Quite unfortunate for them. This is the same way a future AI smarter than us will feel, that it's unfortunate, but for the greater good.
Some people are saying that this might be a sort of law of physics, since the start of the universe complexity seems to increase.
Our destruction might be required even if no harm is meant.
The example given is: "We don't hate ants, but when we need to build a highway we pave over their ant-hills without a second thought".
And once again, if we create AGI and artificial life that is truly more intelligent than us and can exist without a traditional biome, then our danger of going extinct may be irrelevant.
For instance, when cyanobacteria first appeared on Earth, its metabolites were so toxic that it turned the atmosphere poisonous and caused one of the greatest mass extinctions in the history of our planet. What it created was molecular oxygen. The survivors of this extinction were able to use a fundamentally changed biome to harness much more energy in their biologies, leading to more sophisticated and diverse life in the long run. Nature is not benign, malignant, fragile, or judgmental. Nature is persistent.
Creating AGI may very well be an event like that. A mass extinction that nonetheless increases the survivability and diversity of life. It sucks for us, but who are we to dictate the destiny of evolution and the nature of life? Where would we be if the cyanobacteria could decide not to start producing oxygen?
I don't have evidence in terms of AGI, but there is evidence that ants have survived a myriad of changes and disasters in the past (Goggle tells me they evolved 92 mya). So AGI and/or humans would have to do something radically different to the biome for organisms like ants to go extinct. Something not seen since the rise of multi-cellular life. Seems much more likely human civilization bites the dust first.
> Where would we be if the cyanobacteria could decide not to start producing oxygen?
But they didn't have a choice. We do. Why would I care about the possibility for some more sophisticated life form in the far future if it means my species goes extinct?
If we someday hypothetically grasp the awesome power to create an AGI which is better than human, we may have a very real and ethical obligation to do so, even if it is a risk to ourselves. At least that's how I see it. An existential risk is cast under a different light if it also creates a new existence.
In any case, AGI is probably still a lot further away than we think. I feel that problem is going to be like fusion power. It's going to be ~15 years away for about 150 years, or who knows how long. This is a hard problem with monstrous complexity and many hidden, unknown factors.
But I do find it fascinating that in some future time there will be NIMBYs of consciousness fighting against the creation of a better being than us, for very understandable reasons. Yet ultimately it may be our greatest achievement to someday make ourselves obsolete and hand the future over to our offspring, as we do each generation, but in a profoundly different and more final way.
The reason we have complex multi-cellular life forms on this planet is because the environment was stable enough to allow such a thing to happen. Bacteria and viruses might be the only life on a majority of inhabited planets.
Don't get me wrong, I'm not talking about mushroom clouds and some kind of machine uprising, the consequences will be far worse and more insidious. We'll see completely new levels of manipulation, oppression and surveillance instead. Any kind of tool will be abused, this tool might turn out to be too powerful for us to handle.
Thus it become necessary or desirable to create adaptive software (such as deep learning systems), that automatically deals with situations in a fashion that constructed or rule-based systems can't easily deal with (or when it would expensive to create such software or when the model isn't known).
Especially decision support seems to be important here - the parole rating program and medical diagnosis software systems, for example, aim to mobilize the huge bulk of data an organization (or all humanity) has to make proper decisions whereas any human making decisions here is going to have a limited viewpoint once the store of data reaches a certain size.
The problem of these systems still being fragile and difficult to maintain, however, means that the problem of increasing complexity still remains. So in a lot of ways, robust AI would be a hope for a way of prevent civilization from essentially collapsing under its own weight.
Whether this is realistic is another question.
This is not a small problem. Thinkers such as Joseph Tainter locate the tendency of civilizations to collapse in the tendency of specialize to increase and cease to be adaptive to a society.
Actually, I think I can answer this- it's the Singularitarians who promote this idea. Unfortunately, their fantasies have tained the meaning of "AGI", making it synonymous with an evil (or amoral etc) paper-clip maximiser and the like.
A much more realistic sort of AGI, on the other hand, one that we could turn on and off as we pleased, would be a great tool in understanding our own, natural intelligence. Or it might be the case that we'll only ever achieve AGI once we have a better understanding of our own intelligence.
In any case, there's really no reason to assume that AGI will necessarily replace humanity.
But my reaction from a gut level is, what's the value of human life if machines do all the important stuff for us, while also surpassing us in creativity, insight and discovery?
But why would they? There is nothing inherent in the idea of AGI that says they absolutely have to.
This is again the fallacy of the Singulartiarians, that once you have a very powerful piece of technology, it will take over the world and oppress mankind. Why does that have to be the case? Well, it doesn't- and if it looks like it's how historically things have played out, with other technologies, then maybe the fault is not with the power of machines of all sorts but with the mismanagement that comes with greed and its excitement by useful technologies.
In any case there is still nothing that supports the equation AGI = human oppression, right out of the box.
This is a research problem.
Your phrasing makes it sound like you think “Singularitarians” choose to believe that rogue AIs are a potential risk, rather than deriving that knowledge.
Those people are fantasists, at best, charlatans at worst.
Are there benefits of multiple artificial minds? Are they all perfect copies or do they have unique perspectives?
Younger generations eventually succeed older generations as they die off. If we create something better, wouldn’t that just be a better new humanity to succeed the old one?
Second, when we "want" human minds in a possessive industrial sense, it is almost always for a small fraction of their capabilities. (Thank God our need to classify images does not seem to explicitly require servile machines with, say, the ability to love.)
When I was studying pre-deep learning statistical ML in the oughts, it was quite clear that the ML Renaissance was made possible by ignoring romantic ideas about human or expert level performance, and choosing operational criteria around improvement at specific tasks. People building PGMs or complex MCMc models rarely discussed whether larger versions of their models could think. Perhaps early uncertainty about how some DL methods worked opened the door to magical thinking about the possibilities of massive nets.
I think AGI should be thought of as an artistic or poetic interest, which can align with and comment on scientific progress, but is not actually part of the main research dialogue.