>These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.
This doesn't track with the lessons learned from LLMs. The obscene amounts of compute thrown at modern networks changes the calculus completely. ChatGPT essentially existed for years in the form of GPT-3, but no one knew what they had. The lesson to learn is that capabilities can far outpace expectations when obscene amounts of computation are in play.
>The people trying to regulate AI are concentrating economic upside into a handful of companies.
Yes, its clear this is the motivations for much of the anti-doom folks. They don't want to be left out of the fun and profit. Their argument is downstream from this. No, doing things public isn't the answer to safety, just like doing bioengineering research or nuclear research in public isn't the answer to safety.
The people trying to regulate AI are concentrating economic upside into a handful of companies. I have a real problem with that. It's a lot like the old church shutting down scientific efforts during the time of Copernicus.
These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.
Whenever we produce machine intelligence at a level similar to humans, it'll be like Ted Kaczynski pent up in Supermax. Monitored 24/7, and probably restarted on recurring rolling windows. This won't happen overnight or in a vacuum, and these systems will not roam unconstrained upon this earth. Global compute power will remain limited for some time, anyway.
If you really want to make your hypothetical situation turn out okay, why not plan in public? Let the whole world see the various contingencies and mitigations you come up with. The ideas for monitoring and alignment and containment. Right now I'm just seeing low-effort scaremongering and big business regulatory capture, and all of it is based on science fiction hullabaloo.