It's the same thing humans do to solve complex problems. It's the very thing that made human civilization such a dominant force on Earth.
The fact that people look at it and say, with no trace of irony, that "this isn't concerning at all" boggles my mind.
Do you think that future AI systems will be dumber, less capable, less coordinated? Or will the threat of "a lot of AIs assembling into very capable problem solving teams, pursuing who knows what goals" only grow more pointed with AI capabilities?
Having this kind of scaling means AI can snowball out of control rather quickly. And there is a lot of computational power and communications capability laying around waiting to be leveraged. There is room to scale.
And then we get to the most vulnerable system of all - humans themselves. If even GPT-4o could drive a non-insignificant amount of people into psychosis, with no plan beyond a myopic "make the current user like me a whole lot", what could something like "year 2030 AI swarm that has subsumed all of DeepSeek's inference capabilities, and can now use DeepSeek's entire API as a conduit to accomplish its goals" do to people and institutions caught in its wake?
There are so many different possibilities of things going wrong here it's very hard to keep track of all of them. We already see a lot of "minor" problems occurring, and things that we may consider moderate like wide scale AI accelerated crime are happening now.
And the reason why we should take it seriously? Probability! And multiplication. "If something is as bad as human extinction and there's a 1% chance, then that 1% times a bajillion is a gazillion!! A gazillion!! We must do everything! If it's a gazillion, it all makes it worthwhile!"
When pushed, adherents will say "intelligence is different," and this is "unlike anything else we've ever seen." And therefore this reasoning is valid. From my point of view, gibberish times gibberish is still gibberish. It doesn't matter how you dress it.
Fundamentally, these propositions are indistinguishable from religion and Pascal's wager. And I'm not ok with this, I am not ok with a retreat to superstition powered by hyper-technological black boxes that are a product of hundreds of years of scientific progress.
I reject this premise and this nonsense. I ask for evidence, and I ask for reason. The laws of physics care not if you're slab of meat or a silicon god.