Second, if you are truly “accepting the premise” of superintelligence, a superintelligence would know exactly this line of reasoning and just opt to at least mimic vulnerability to prompt injection.
I wouldn’t hang civilization on this proofpoint. Doesn’t seem meaningful at all.
And I wouldn't risk, or slow down getting to, the potential benefits of developing AI further, which are tangible and measurable, because of vague threat scenarios with little to no evidence or methods of measurement that seem like good plots for a SciFi B-Movie.
Pascals wager, as an argument, relies in no small part on the assumption that to believe in a god-like entity doesn't come at a significant cost. Slowing down or abandoning the development of AI however, does.
AI optimists tend to conflate the two for some reason.
I am using this as an accessible (in term of discussion material) hallmark for the ability of the system to self improve. Accessible because everyone has heard of it by now, and so I don't have to spend time explaining it.
The AI Doomsday scenarios require that a system self-improves massively, even beyond our ability to even theoretically understand. After all, some of the assumptions give them next to magical abilities like nanotechnology that we similarly don't know if it is even possible.
It stands to reason that an entity that can do that, or is in the process of becoming capable to do that, would begin by eliminating obvious flaws in itself, that would make it comparatively easy to stop.
After all, it's not much good being a super-intelligence, if some smartpants with a laptop and too much time on his hands can just trick me into deleting myself, is it?
> But it seems to me that von Neumann or Einstein was just as vulnerable to getting hit in the head with a baseball bat as anyone else.
Yes, and despite both of them being geniuses by human standards, neither of them was a superintelligence on the level the common doomsday scenarios ascribe to AI.