You don't have bunch of cold blood killers going around not just because police is so good and killers are dumb and need AI help. It's because you live in functioning state where society have enough resources so people happy enough to instead go and kill each other in Counter Strike or Fortnite.
I totally agree that AGI could be a dangerous tech, but it's will require autonomity where it can manipulate real world. So far GPT with API access is very far from that point.
The danger also doesn't need to be an AI generating code to hack the Gibson. It could also be things like "how do I manipulate someone to do something". Asking an AI for a marketing campaign isn't necessarily amoral. Asking it how to best harass someone into committing self-harm is.
As Nathan states:
>And further, I argued that the Red Team project that I participated in did not suggest that they were on-track to achieve the level of control needed
>Without safety advances, I warned that the next generation of models might very well be too dangerous to release
Seems like each generation of models is getting more capable of thinking, beyond just regurgitating.