Re:OpenAI illegally barred staff from airing safety risks, w...
by ben_w on Tuesday July 16, 2024 @10:48AM
(#40977133)
Before it can or before it has more than 50% chance of having actually done so? Because those are different things.
For the first, you don't need AGI, no matter which of the many definitions you have for what that means. The automation used by the US for their early warning radar in Thule Site J was not programmed to know that the moon does not have an IFF transponder and that's OK, while the Soviet satellite-based system was triggered by reflections of sunlight.
Likewise covid, cancer, smallpox, ebola, HIV — these things do not need "general" intelligence, they're self-replicators that hijack our bodies and have no goal besides what they do.
Some talk of "P(doom)", but (and I should blog about this) every example I have seen has a blind spot that is especially embarrassing given the circles in which this discussion proliferates is very familiar with Bayes' theorem: P(A) by itself isn't meaningful, it's P(A|B), so when it's doom, doom given what? There's too many options for what each of technology and politics will allow AI to "look like". Doom within 24 hours because an AI takes an instruction literally and without regard for our ethics, is very different to 'doom' in a million years caused by natural extinction that is never prevented because every single time we make an AI we keep finding that it builds a spaceship and flies off into the void so it doesn't need to deal with us any more.
Me, I think that when LLMs alone (again, don't need to be "AGI" by any particular measure) have the capabilities of someone with 5 years post-graduation work experience in just biology (or similar subject with similar potential for accidental harm), then we've got a reasonable chance of multiple incidents each year where idiots (and/or misanthropes) make a Covid-scale pandemic (or, if the LLM is a software rather than biology helper, a deliberately corrupting rather than greedy version of the current encryption-blackmail malware) — and that kind of scale has a combined risk of about 10% of one of the incidents being of sufficient scale that global society collapses and doesn't recover, before it's happened so often that everything gets locked down, possibly including a Butlerian Jihad. And if you think humans would "obviously" take steps to prevent this after the first failed attempt, I would point out quite how many US politicians have been shot and yet those same politicians still refuse to consider that the 2nd Amendment might possibly be not that good. (In case you think this is topical: https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_... )
Remember, the AI doesn't need to actively hate you, it just needs to do something dangerous — and it doesn't matter if that danger comes from a long-term plan that requires it to engage in amoral power-seeking, or a short-term plan from a deranged monster of a human using it, and even when both it and the user are "pro human" it can still do this from simply being unaware that one part of the plan is incompatible with life.
For the first, you don't need AGI, no matter which of the many definitions you have for what that means. The automation used by the US for their early warning radar in Thule Site J was not programmed to know that the moon does not have an IFF transponder and that's OK, while the Soviet satellite-based system was triggered by reflections of sunlight.
Likewise covid, cancer, smallpox, ebola, HIV — these things do not need "general" intelligence, they're self-replicators that hijack our bodies and have no goal besides what they do.
Some talk of "P(doom)", but (and I should blog about this) every example I have seen has a blind spot that is especially embarrassing given the circles in which this discussion proliferates is very familiar with Bayes' theorem: P(A) by itself isn't meaningful, it's P(A|B), so when it's doom, doom given what? There's too many options for what each of technology and politics will allow AI to "look like". Doom within 24 hours because an AI takes an instruction literally and without regard for our ethics, is very different to 'doom' in a million years caused by natural extinction that is never prevented because every single time we make an AI we keep finding that it builds a spaceship and flies off into the void so it doesn't need to deal with us any more.
Me, I think that when LLMs alone (again, don't need to be "AGI" by any particular measure) have the capabilities of someone with 5 years post-graduation work experience in just biology (or similar subject with similar potential for accidental harm), then we've got a reasonable chance of multiple incidents each year where idiots (and/or misanthropes) make a Covid-scale pandemic (or, if the LLM is a software rather than biology helper, a deliberately corrupting rather than greedy version of the current encryption-blackmail malware) — and that kind of scale has a combined risk of about 10% of one of the incidents being of sufficient scale that global society collapses and doesn't recover, before it's happened so often that everything gets locked down, possibly including a Butlerian Jihad. And if you think humans would "obviously" take steps to prevent this after the first failed attempt, I would point out quite how many US politicians have been shot and yet those same politicians still refuse to consider that the 2nd Amendment might possibly be not that good. (In case you think this is topical: https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_... )
Remember, the AI doesn't need to actively hate you, it just needs to do something dangerous — and it doesn't matter if that danger comes from a long-term plan that requires it to engage in amoral power-seeking, or a short-term plan from a deranged monster of a human using it, and even when both it and the user are "pro human" it can still do this from simply being unaware that one part of the plan is incompatible with life.