Just the threat of a malicious AGI that people take seriously enough would cause ourselves to pull the plug on the Internet and all computers until they can be disinfected and new filters put in place.
It doesn't matter if the AGI is capable of actually inflicting damage or not.
Germany, France and other large countries became virtual ghost towns overnight as it became illegal to leave your home for other than critical errands.
Wasn't it the same in big parts of the US? I remember seeing videos of a helicopter flying over LA without cars or people out on the streets.
But sure, the whole world wouldn't coordinate to shut down.
AI risk enthusiasts tend to define "bad goal" as one that sounds reasonable but has unexpected side effects. The real risk to infrastructure and physical safety though is that people will use AI to assist in hacking during a hot war. In this case the goal is only bad from the perspective of one side of the conflict, and the solutions are the same in any event because human hackers can also do a lot of damage, at least in theory. Our computing infrastructure hasn't ever really been tested in a hot war between sophisticated powers, the closest is currently Russia v Ukraine where electricity outages and the like have been caused by hackers but the damage seems minor compared to that inflicted by conventional warfare. So this is maybe a "hopeful" outcome - the much touted cyberwarfare has fizzled so far.
So the argument would have to become that AI hackers will be vastly better at disruption than human hackers are. And I guess I can sort of believe that in principle one day but currently they are very far away from this sort of capability.
I think to say with confidence that we're doomed or that there is no risk to the AI system currently in development would be intellectually arrogant. I'm not even convinced you necessarily need AGI before AI becomes a risk to humanity. An advanced enough auto-GPT bot given a bad goal might be all you need. Cripple enough infrastructure and humanity is going to be starving to death within weeks.
Personally I think humanity is probably okay for another 5 - 10 years, but after that I have almost zero certainty. It seems to me that it's quite possible we could have AGI by the end of the decade if we get another breakthrough or two.
Even near-term I have no strong conviction. I have a decent amount of experience building and working with neural nets, but my experience is so far from scale and complexity companies like OpenAI are working at my ability to predict even what might come out in the next 1-2 years is very low. I'd basically just have to assume the next GPT is going to be somewhere between 10-50% jump from what we got this year. But what if it's 200% and the trend is moving exponentially?