From the perspective of AI Safety (may get important soon), it is perhaps a very bad idea to allow AI agents to suggest execution/injection of arbitrary code into random places with a review from a single human that needed help from Stack Overflow.

[And don’t believe ChatGPT that claims it is an only a Language Model. It is not. It is a RL Agent trained with PPO.]

That's a fair point. Does the same argument apply to GitHub Copilot / ChatGPT?

Does it apply to things that aren't programming? e.g. People are already using these AIs for legal work.

One of the first steps of a runaway AI attempting to secure its presence would likely be inference/GPU/TPU compute access. And code injection is a vector. There are multiple other vectors.

But, perhaps considering AI as an adversary is a bad idea. And alignment along the “Love is all you need” lines is the actual solution. Tricky problem…

I was laughing last night when I realized that even if you want to make an AI that only knows and cares about maximizing paperclip output, you also need to teach it to love all humans unconditionally to prevent the apocalypse.

> You could just make the AI answer private to the asker.

So then the asker gets the wrong answer and no one else can correct it? Seems even worse.

That doesn't solve the problem that GPT-based AIs don't "know" when they're wrong. They will happily spit out authoritative-looking nonsense when presented with questions they don't know how to answer -- and the people asking the questions may not be able to recognize those answers as wrong.