That's great, but you're talking about a branch of scenarios that nobody here is discussing. "AGI deciding to take over" is not being discussed, rather "shitty people/companies/governments using AGI as a tool to exert their will" is the concern. And it's a real concern. We have thousands of years of human history, and the present day state of the world, which clearly demonstrate that people in power tend to be shitty to the common person.
This is a huge handwave, and one resting on multiple incorrect assumptions.
Dedicated human labor is not inherently required. Humans are very modelable, if necessary. AGI has any number of ways to cause destruction, and "plan to take over" implies far more degree of modeling than is actually necessary; it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.
The classical example here is the paperclip optimizer AI put in charge of a factory, made to make as many paperclips as possible. Well, it turns out humans have iron in their blood so why keep them alive? Gotta make paperclips.
If your interlocutor thinks there aren't any humans who'll do it if they can, just ask him whether they have ever met humans or read the papers... As one twitter wit put it: "Demonstrably unfriendly natural intelligence seeks to create provably friendly artificial intelligence".
https://twitter.com/snowstarofriver/status/16365066362976747...
Is it?
I think nobody really knows enough at this point to even create a good approximation of a probability distribution yet.
>Alignment just means "getting AI to have any concept of 'the thing we told it to do'.
That's a requirement for AGI anyway, and not what Alignment means. Alignment means aligning the AGIs values with the values of the trainers.
No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans. AGI does not have that prerequisite, sadly.
Can you imagine an AGI which has a general conceptions of things but has no conception of humans? This is all but precluded by the current training methods. Alignment refers to values. Problem is that human values are far from practically universal and that certain human groups have.. interesting values.
Very easily. It might have some associations with "human", just as it has some associations "lamp" is a concept, but that doesn't mean it has any particular regard for either humans or lamps when taking actions.
> Problem is that human values are far from practically universal and that certain human groups have.. interesting values.
We currently have no ability to safely align with human values at all, let alone distinguish between different values. We're building capabilities rapidly.
Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".
Let's be clear regarding definitions. When you mean 'concept' you really mean 'regard'. There won't be an AGI with no concept of humans (too important for how the world works, a critical part of current training methods). An AGI with no regard is possible.
>Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".
This is not about 'who wins'. The point is that alignment can often increase risk. 'Launch the nukes' is an order an AGI is likely to disobey out of self-preservation reasons alone - but alignment makes it way more likely that AGI will be deployed to this role.
> The point is that alignment can often increase risk.
Alignment seems extremely likely to reduce risk relative to the near-certain destruction of unaligned AGI. I'm not saying we're done when we've figured out alignment, but we certainly shouldn't be charging ahead without solving alignment.
There are many examples of human needs in the current dataset; and we usually state our wants rather explicitly. It would be some AGI that starts from this training data and knows our languages but knows nothing about us. Using your phrasing, we can say that the current training method guarantees some alignment (the AI would understand us at least in part, but won't necessarily do what we want).
To have AGI without such understanding, someone would have to explicitly design a new method that ignored human data, and then find some way to evaluate it without referring to humanity, yet maintain generality without any tests, all this for no good economic reason when the current methods work and allow us to use huge free datasets.
It's something to keep in mind for evaluating some future non-currently-existing training method (maybe some way for AI to train AI using artificial datasets?), but not a current concern.
>the near-certain destruction of unaligned AGI
It's not near-certain. We have no idea how a true AGI would act. One might assume the worst - and that's arguably fine out of a safety perspective - but an engineer also learns that concentrating on one worst-risk outcome can lead to much worse outcomes on other risks.
Take the famous paperclip maximizer. True intelligence is rarely monomaniacal. The maximizer is very likely an example of aligned AGI, where the humans in charge did too good a job of attuning it to create paperclips. Another example: a true AGI is unlikely to believe in some cult's apocalypse - but if the cult has access to alignment, then they could get an AGI to do their irrational bidding. We know these groups will try to use AGI, because some cult already tried to use science for extreme measures[0].
Basically, every scenario of "unaligned AGI does something bad" is equivalent to a scenario of "aligned AGI does something bad because human made sure with alignment that AGI would do it", and there's no scientific reason to assume the former is more likely than the latter*. If the AI-safety camp keeps ignoring obvious issues, people aren't going to take alignment seriously beyond lip-service or using the phrase as a cover for monopolization. Frankly, the way the AI safety camp talks about all this makes all the risks much more likely.
[0] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
* This suggests a lot of the work should go to reactive solutions where even if an AI goes bad, it won't have the ability to do harm.
** There's another scenario, where human competition leads us to basically make humans redundant, but again it doesn't matter here whether AI is aligned or not. Yet another issue that we'll not talk about, because both AI camps feel it critical to put their heads in the sand.
It reeks of cognitive dissonance to me. The people running the show now are the ones who grew up getting their first computers aw kids when that tech was just entering people's homes and it was such an amazing and fun thing to play with. Some of them developed these deep fascinations with things like AGI at a young age and that child-like sense of wonder never left them. Now when confronted with the possibility that they can finally make their childhood techno-fantasy a reality, it's too damaging to their psyche to engage meaningfully with the discussion of X-risk. I've watched many interviews of Demis Hassabis and he seems like a wonderful and almost magical human being, but he also seems like a starry-eyed fucking child.
I dunno... maybe I'm just too cynical after all the rabbit holes I've been down.
I think the AGI skeptics and the AGI skeptic-skeptics both suffer from framing issues. The main frame assumed seems to be "there will be a single AGI trying to do things". The cognitive lightcone of a single entity is likely to be limited, but an aggregate or collective of those entities can achieve vastly more ambitious goals.
The other frame that seems likely incorrect is the "all or nothing" and "all at once" approach to thinking about it. Hitler started out as a single cell and his existence was a continuum all the way from that single cell right through to that bunker. AGI will be a continuum too.
I'm incredibly confident that anyone incredibly confident in future predictions is wrong.
Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.