Second, I wonder if we could force conversation-points where AI talks to other AI's to be simple and perhaps open-source.
So a AI banking API talking to the Tesla API can never say much more than "yes this driver has funds" etc. It can't talk about world domination.
And even if we were hypothetically successful, and has been demonstrated already with the release of primitive LLMs, someone out there will build unaligned AI purposely.
I was just thinking about North Korea, and how AI will enable statewide surveillance on a whole new level, but then the thought popped into my head that maybe if AI can't be stopped, it can't be stopped there either, and maybe it'll be benevolent, and in 30 years they will be freed. ¯\_(ツ)_/¯ Hard to know how this plays out.
Hoping it will be benevolent is about the only thing we have at this point. Many, but not all, alignment safety researches believe that it will inherently exhibit properties that make for disastrous outcomes. Since they have no solution for this problem if it exists, at this point hoping they are wrong is the best option.
I agree. There is a very large moat between where we are now and the envisioned AI that people hope for. And that moat is filled with lots of problems without any clear answers.
> because AI will be better than us at everything
Yes, I don't think this concept still has not been completely digested by most people. AI is essentially a skill/technology replication machine. It is going to bring about a crisis of meaning in society as well.
> The question will be if we have safety mechanisms if AI starts going rogue.
I've spent a lot of time thinking about and writing on this topic as well. Under the current premise of alignment theory, I believe it is a complete logical fallacy or paradox. I don't see it as a hard problem, but rather an impossible problem.
Now, this is just from the perspective of taking alignment theory's own proposed conditions. That being AI will be power seeking, scale infinitely on its own and we will solve this by applying humanities values.
I explain why I perceive this as an impossible solution here.
https://dakara.substack.com/p/ai-singularity-the-hubris-trap