The risk of poorly aligned AGI as described is real, makes sense to try to work on the problem before we need it (when it’ll be too late).
If you have some actual position of substance, articulate it or just don’t bother replying at all.
Claims made without evidence can be dismissed without evidence.
Either you think AGI is impossible for some reason or you think it is possible, but the goal alignment issues don’t matter. You haven’t given any explanation for either. Maybe you think something else? I wouldn’t know because you don’t say anything.
Obviously there isn’t “hard scientific backing” since AGI doesn’t exist yet. There are still unknowns about what would be required. That doesn’t mean it’s impossible to think about what its existence could mean.
It was possible to think about the consequences of flight before airplanes existed.
I think AGI is possible because we’re surrounded by general intelligence and there’s nothing magical about brains or biology. It’s just the timeframe that’s unclear.
We already see issues with poorly optimized reward functions on a smaller scale, both in ML models (classifying based on unintended things) and in humans (drug addiction).
Like the anthill destroyed by the construction of a dam. The humans don’t hate the ants, aren’t putting them in zoos, they just don’t think of them in pursuit of their goals. In the AGI case it’s even possible for those goals to be “dumb” or pointless if we mess up the reward function or some unanticipated thing fits it (paper clips).
The risk with a self-improving AGI is that it is way smarter than us and it can think way faster. In my experience people have a hard time really visualizing this.
Imagine a human trying to trick a chimp (or an ant) into doing something so the human can pursue its goal. The gap between a human and an AGI would be even larger than that.
Imagine scaling up a regular human brain’s operations per second a billion times so it’s the same architecture but thinks that much faster (no longer constrained by bio energy or head size). Humans wouldn’t be able to respond because to the machine it’d be nearly like interacting with a static object, it’d be trivially able to think/act around you.
That’s why the research around AGI is about the goal alignment problem. Which is hard because humans are not perfectly aligned either. It’s also why “just unplug it” is dumb.
The optimistic outcome if the alignment problem is solved is enormous intellectual capacity to solve any problem we might have.