It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes.
But why the hell would you do that?!?!
ChatGPT can already write code. A magpie cannot do that.
And it’s a powerful tool. Even staplers have rules around their use: no stapling people, no hitting people with a stapler, don’t use a staple to pick a lock, etc.
But nobody blames the stapler, is my point.
Reduced to the sum of its parts, the internet is less than a magpie, yet viruses and contagion of many forms exist in it, or are spread though it. ChatGPT 2.0 greatly amplifies the effects of those contagion, regardless of our notions of what intelligence or agency actually is.
There are limits, but even if there weren’t, we’re no closer to AGI today then we were a year ago. It’s just a different thing entirely.
LLMs are cool! They’re exciting! There should be rules around their responsible operation! But they’re not going to kill us all, or invade, or operate in any meaningful way outside of our control. Someone will always be responsible for them.
> But why the hell would you do that?!?!
Exactly. Except that they are doing it exactly, and we are the ones asking 'why the hell would you do that?!'.
A chatGPT 2.0 knows everything, really. If you command it (or it reaches the conclusion itself for some unrelated commands), it can hack nuclear arsenals and nuke everything, or worse.
Pretty much everything here is incorrect. An LLM is not omniscient. An LLM does not think or reason. An LLM does not reach conclusions.
There is no "AI" here in the sense you're saying.
On the other hand, we all know it’d be GraphQL…
And GPT already knows which buttons to push. It takes a little bit of prompt engineering to get past the filters, but it'll happily write inflammatory political pamphlets and such.
But in short, as I said in my GP comment, systems like OpenAssistant are being given the ability to make network calls in order to take actions.
Regardless of whether the system "knows" what an action "means" or if those actions construe "harm", if it hallucinates (or is prompt-hijacked into) a script kiddie personality in its prompt context and starts emitting actions that hack external systems, harm will ensue.
Perhaps at first rather than "launch nukes", consider "post harassing/abusive tweets", "dox this person", "impersonate this person and do bad/criminal things", and so on. It should require little imagination to come up with potential harmful results from attaching a LLM to `eval()` on a network-connected machine.
"Aligned" is a term of art. It refers to the idea that a system with agency or causal autonomy will act in our interests. It doesn't imply any sense of personhood/selfhood/consciousness.
If you think that Bing is equally autonomous as a stapler, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people (that's not hyperbole, I mean that literally, indeed full extinction of humanity is a plausible outcome too). A stapler is understood mechanistically, it's trivially transparent what's going on when you use one, and the only way harm can result is if you do something stupid with it. You cannot for a second defend the proposition that a LLM is equally transparent, or that harm will only arise if an LLM is "used wrong".
I think you're getting hung up on an imagined/misunderstood claim that the alignment frame requires us to grant personhood or consciousness to these systems. I think that's completely wrong, and a distraction. You could usefully apply the "alignment" paradigm to viruses and bacteria; the gut microbiome is usually "aligned" in that it's healthy and beneficial to humans, and Covid-19 is "anti-aligned", in that it kills people and prevents us from doing what we want.
If ChatGPT 2.0 gains the ability to take actions on the internet, and the action <harm person X> is the completion it generates for a given input, then the resulting harm is what I mean when I talk about harms from "un-aligned" systems.