You can say the same things about human societies and nature. Both can be extremely dangerous and all the things you've said about complex computer systems apply to them.
> it's extremely successful at swallowing the world and convincing us to feed it more and more CPU, RAM, disk, data.
because IT (what a pun) adds a lot of value.
> But it will win.
I don't know what you mean by that. It's still a tool, although we give up some control so it can manage a part of our society. Historically, this was a task for the elite (it still is) - a bunch of people who had some privileges. Most give up some control or freedom so that the elite can manage the society. If things went bad, the masses were able to defend themselves most of the time (guillotine and good ol' pitchfork) - Marx talks about this eternal cycle.
I would rather worry about elites that build immense power through technology - today we have tools of mass manipulation and destruction. I don't fear intelligent machines, I fear that humans use those tools to entrench their power. I think this ambiguous fear of technology you're describing is too vague - it sounds more like mild technophobia.
You can say the same things about human societies and nature."
You can, but the thing is, we already know a lot about how that works. We've evolved into it over millions of years. We have no idea about this new stuff.
You want to talk about about small groups of 100-150 or so people work together to accomplish something? I've got hundreds of books, stories, history, archaeology. We have the wisdom of the ages. You want to talk about how 20 thousand semi-intelligent agents interact with billions? We have nothing aside from endless optimism and the fun of making stuff. (Doesn't make it bad. It means we should grow the hell up and be a bit reticent about what we're mucking around with. Make smaller mistakes more quickly would be my advice.)
I don't really believe in a singular entity in there, other than a convenient abstraction for now. Rather there are many different processes that are all optimizing this and that. The sum is essentially intelligent--it processes information, makes decisions. Those decisions help it grow and spread and become more efficient. These optimizers don't speak human languages. They don't necessarily think in abstract concepts like us. Maybe they are just tables of numbers optimizing themselves, maybe they are like standing waves in a sea of numbers, like our thoughts are standing wave patterns in the electrical activity of our brains. It's hard to express because our words fail us.
Problem is, the whole damn thing is so complicated that we cannot know anything really definitive about it. The substrate is there. The soil is fertile. The capability is there. And so many forces are aligned to feed more and more resources into the whole thing. Across the board, essentially the only thing we can come up with is to upload all of our data into this massive cloud thing and use computation to do something with it. That is just like injecting raw materials and energy into the fabric. And those who do this successfully are rewarded financially. So all forces point to more of it.
Whatever that huge mesh is, it's extremely successful at swallowing the world and convincing us to feed it more and more CPU, RAM, disk, data. It keeps convincing us to hook it into more things to sense and control--more cameras, phones, drones, markets, even weapons.
Thus in some abstract sense, it is already here. And we can't get rid of it. Can't shut it off. We're dependent, and it's parasitic--for now. Yet every incentive we have right now is to make it more autonomous, more robust, more powerful, and give it more and more control. It works best when it is more centralized.
Teeth or not, conscious or not, this computer thingy is gobbling everything. Do we control it? Clearly not. Can we predict what it will do? Not even close. Is it good for us? Who knows? But it will win.
Well played, computer thingy.