It's possible there were alternative outcomes where everyone won. A more fair society could have found a way to make sure the replaced textile workers benefited from the machines that replaced them.
If we make machines that replace all of our economic roles, then it's possible we could make it so we benefit too, but with AGI it's not nearly enough to just choose to use it that way. AGI will not be aligned with our values if we don't make it that way. If we solve the alignment problem, then we would be able to make AGI that shares our values. AGI could solve our problems, or uplift us to its level, if we accomplish this. But this is not the default outcome if we don't solve the alignment problem.
While I am worried about it, I also recognize that worrying about it doesn't really do anyone any good -- the best thing I can do is to keep growing my skills as a tech person so that I can be as prepared as is reasonably possible when it eventually does arrive. Heck, maybe I can actually contribute to helping align it positively with humanity, I don't know. The point is that there's no point in stressing too much about it.
one would expect that initially, agi will simply act in the existing role we have already established for artificial persons - the corporation, notably uninterested in human well-being.
AGI will, as I said, make many things possible. It will be possible to keep a human being alive forever and suspend them in a state of pure pain for eternity. It will be possible to extract memories and information from the human brain. It will be possible to control human behavior with invasive brain surgery. Imagine every grotesque thing that you can do to a human being — it’s on the table for AGI enabled governments in the early days if that’s how it plays out. And that’s just one slice in the infinite pie of grotesque possibilities. You can’t just opt out of these things if they provide a material advantage to your country/meta organism.
Biological brains are incredibly complex and if that level of complexity is needed to host intelligence I think we're a long ways off from building any true AGI, even if we pumped every bit of engineering skill our society has at it. And even if we _can_ build AGI I doubt we'd have the resources to host more than a handful of them for years to decades after their creation or that their intelligence would be anywhere near small mammal level.
(It might be tempting to imagine this from a higher-level perspective and assert that each of these successors were somehow more interesting or had more fulfilling lives than what they replaced, and so therefore these successions might be ultimately good in the long run, but that's not meant to be part of this analogy. We won't have some genetic lineage with AGI; it will probably be more alien to our mind's design than actual aliens produced by evolution like us would be. The AGI might not build a world replacing ours where many of its kind have lives we would find fulfilling or interesting. It might care just about making the world completely predictable and safe, empty of outside threats and life, and hibernating in a place of safety. Plenty of animals live solitarily and might do this if they were naively uplifted into superintelligence. Humans and social animals instinctually believe in the goodness of friendship and society because evolution saw we were good at surviving in groups and molded our instincts to encourage us to do that. AGIs won't necessarily have that if we don't design that into them (or set up the process that designs them to do that).)
I will make everything very simple. All of AI industry and research boils down to one thing: mining algorithm space. We set up programs that search algorithm space automatically until they find one that demonstrates desirable behavior. And all the progress in recent years boils down to this: we have hit veins of intelligence in algorithm space that have exceeded what we thought possible.
If you think you know what’s going to happen next because leading edge algorithm X has properties Y and limitations Z then you are lost. We are mining algorithm space and we keep striking veins. Money is being poured into mining and it is a fact that we will keep hitting veins.
General intelligence is probably a broad category. If you were to define it mathematically, there is probably more than one mathematical kernel that can underpin a generally intelligent algorithm. So the number of algorithms that are generally intelligent are probably much higher than intuition would lead you to believe. Eventually we will strike a GI vein.
Again, if you think AGI will do X or do Y then you are lost. It will only do one thing: it will change the economic equation of life on this planet drastically and permanently. It will be the worst thing that has ever happened in terms of human well-being.
If we took control of the fabs and enacted limits on processor feature size we could slow the mining enough to find a final solution. It sounds silly until you realize that it’s truly the world at stake.