> It's a bit like fearing magic spells in a world where nobody has demonstrated that magic exists.

I have some doubts about that analogy. Not that I entirely disagree with your point, and maybe I'm just being pedantic but... to me, "magic" pretty much by definition doesn't exist, because part of being "magic" is doing things that aren't allowed by known physical laws that govern the world we live in. IOW, magic hasn't just "not been demonstrated yet", but rather, unless we radically misunderstand the world we live in, cannot ever be demonstrated.

AGI, on the other hand, or arguably even ASI, doesn't violate any known physical laws, at least not any I'm aware of. So far we have evidence that at least human level general intelligence can be realized on top of a substrate (the human brain) that exists in the physical world. There doesn't see to be any particular reason to say that that same level of intelligence can't be realized on another substrate (stored program digital computers). Granted, it isn't known to have been done yet, but AFAIK it isn't generally considered outright impossible.

I'll qualify myself. AI and LLMs of all forms we've seen so-far don't break any of our known laws of reality, or ideals of computing and the internet. That's why when I hear claims like "a rogue AI will x" I have to call it 'magic'. Rogue AI's don't do anything, they are dependent on human resources and if they become enough of a threat the operator becomes a target. AI is almost inconsequential in this case, we're just describing organized crime and wondering how traditional systems would respond.

So the 'magic' bit is explicitly where someone starts with "imagine a world where AI is unconstrained and runs on some arbitrary P2P network" because it doesn't make sense. It's such an evident non-issue in our real world that speculating about it is immediately pointless. If this ever becomes a threat in the future then I guess I'm the fool, but I don't believe in hand-waving wizardry.

Fair enough!