Basic human competence is already signing a law keeping AI away from nuclear weapons management. Just letting a LLM have an internet connection has not resulted in any sort of doomsday scenario.

To be fair, nobody said it would. But getting into the habit of being paranoid about hooking a proto-AI up to the internet now would be good, instead of what we're doing, which is happily gathering lots of experience with enabling AI agency as much as humanly possible.

We literally have "Let's help AI destroy the world" projects. (In jest, for now...) We're like the superintelligence version of that Doctor Who race that really wants to be conquered by a strong alien invader.

Those projects are useful for determining the actual capabilities of the AI for malice as they improve. Doomsdayers really tend to overestimate how much can actually be done with one laptop and an internet connection.

Sure, and I fully support ARC's red-teaming efforts. But at this point, we don't need superintelligence - a marginally intelligent AI could break out of our current safeguards.

Also, "how much can be done with a laptop and an internet connection" surely depends on whether you are in fact a program running on that laptop.

Why exactly would any person be a program on a laptop?

I mean, I'm suggesting that the laptop is running a LLM. Not too likely, but maybe this is, say, six years in the future and it's a workstation laptop with a beefy NVidia GPU and also networks have become even more efficient, so you can fit a dangerous amount of agency into 32GB of VRAM.

Calling the behaviour of GPT in these scenarios "an effort" implies a model of how it works that I don't think everybody shares.