I'm pretty sure they have stock arguments against what you said, even from people like Nick Bostrom who I think have a bit better handle on the situation. In that vein though, I think making the world more robust in general might be a way forward. Having multiple broad spectrum anti-virals for every class of virus ready to go and stockpiled, having flexible on-shore manufacturing infrastructure, with supply chains that are on the same continent or at least have massive component stockpiling on national levels, and having smart and vigilant people running these systems from the very top down to the janitors and security guards. Make a world where the machine stopping because a few agents, human or otherwise, doing something unexpected, much less likely. At least to the point EAs can start buying mosquito nets again.

something never happening before == can't happen is extremely dangerous attittude. I suggest you read Taleb and his duck example.

perhaps its more about "worry about the issue when it _can_ happen" as opposed to planning for the expiry of the sun today.

>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions.

This has been discussed, file under "warning shots" https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-sh...

That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.

Was there even a serious policy discussion of banning gain-of-function research?

Gain of function research is supposed to help with pandemics, right? Did it help with COVID?

I don't think I know enough to evaluate the case around gain of function research, all I know is that the blog post didn't make any effort to.

GoF research was banned in the USA for long periods, that's why Fauci was outsourcing it to China and why DARPA refused to find that project.

No, not only has GoF not helped with COVID and probably caused it, but that's also true of the entire field of virology.

Alexander the Great or Napoleon have been limited in how much damage they can do, because they're human too. They don't live for hundreds of years, they can't copy their brains to new hardware, and they can't modify their own minds to remove that pesky conscience when it slows them down. Perhaps most importantly, Napoleon's happiness, power, and continued survival depends on having a civilization of surviving humans working under his command, whereas that might only temporarily be true for AI.

At best, history serves as a counterexample to the idea that if an AI goes bad the attendants would just unplug it, seeing how often it is that dictators don't get stabbed by their aides as soon as they start causing mass deaths, instead often receiving broad popular support as the world burns.