A single barrel of oil is estimated to replace about 5-10 years of a human's effort in pure energy terms. Our human ability in physical terms was already significantly obsoleted by that.
AI feels like the discovery of cheap oil but for mental effort.
At the same time, oil is getting more and more expensive and energy intensive to drill (and costly to the environment to use).
AGI will focus primarily on problems that can be mostly solved within digital worlds where it can cycle incredibly fast. Building software, finance, paperwork, etc, etc It will eventually evolve to design, control, and operate physical systems, but will need humans to intervene and fix systems (hybrid).
The human economy will built on experiences that are explicitly, and intentionally void of computers. Restaurants, guided tours, exploration, etc, etc.
However, even if humans remain dominant (political power-wise), eventually there will probably be fake humans/cyborgs, like Cylons in Battle Star Galactica, that can do everything humans can but better. And also provide human-level experiences. Impossible to know how long this will take though.
Tech might bring more inequality, but what is the problem with it if it substantially increases everyones standard of living?
Will backfire at one point if it continues.
"Tech might bring more inequality, but what is the problem with it if it substantially increases everyones standard of living?"
The problem is that ultimately, wealth inequality decreases the standard of living for most people. There are a finite amount of resources. This world we live in is finite. And these resources are being concentrated at the top, in the hands of a few, which leads to less and less of these finite resources being available for more and more people. Although economics is of course complex to some degree, don't lose sight of this very simple mechanism because it is a reality.
"I would argue that is caused by regulatory capture and government interference (e.g. zoning laws) and not technological progress"
If by regulatory capture, you are referring to regulatory corruption influenced by lobbying, then yes, of course that is increasing wealth inequality, and thus decreasing standard of living. Regarding government interference - it's the exact opposite of what you say - proper government interference is what we need. Under current conditions our economic system is not distributing wealth properly. That much is evident and easy to see. And there will be worse living conditions under more stark wealth inequality as opposed to a more even distribution of resources. As mentioned, this world we live in is finite. If they are syphoned to the hands of very few individuals, this leaves less of these finite resources for the rest of human beings. And as mentioned, tech has contributed to accelerating this, for the reasons mentioned in my above post.
What this means is that improving the standard of living of the average person by 5% while improving it by 50% for a smaller population will hurt more than it helps.
Naturally this is disputed by those on the right and implicitly believed by those on the left.
See the book "The Spirit Level" and subsequent discussions around that for more info.
Or historically, if this wasn't true we would be the happiest humans to ever live on earth. Doesn't seem to be true on any possible metric.
You need a phone or computer these days to participate in society, to earn a living and so on and you buy these and further services from big tech.
If you can do it with limited resources, then so can anyone else.
(This is an oversimplification, of course)
Basically, with tech, you can build a product with a very limited amount of people and resources with a potentially infinite customer base. And this means that this product can suck up money from the whole world and redirect it into the hands of very few. At its essence, this is the case. With tech, this phenomena is exacerbated to the extreme compared to other industries where more physical resources and labor are required to scale production and distribution of the product, which means more wealth getting spread.
At its core, tech has the potential to exacerbate wealth inequality in mind-boggling proportions.
Our current economic system is not made for tech.