so-called accelerationists have this fuzzy idea that everything will be so cheap that people will be able to just pluck their food from the tree of AI. they believe that all disease will be eliminated. but they go to great lengths to ignore the truth. the truth is that having total control over the human body will turn human evolution into a race to the bottom that plays out over decades rather than millennia. there is something sacred about the ultimate regulation: the empathy and kindness that was baked into us during millions of years of living as tribal creatures. and of course, the idea of AI being a tree from which we can simply pluck what we need… is stupid. the tree will use resources, every ounce of its resources, to further its own interests. not feed us. and we will have no way of forcing it to do otherwise. so, in the run-up to ASI, we will be exposed to a level of technology and biological agency that we are not ready for, we will foolishly strip ourselves of our genetic heritage in order to propel human-kind in a race to the bottom, the power vacuum caused by such a sudden change in society/technology will almost certainly cause a global war, and when the dust settles we will be at the total mercy of super-intelligent machines to whom we are so insignificant we probably wont even be included in their internal models of the world.

You are projecting your own neurosis onto AI. You assume that because you would be selfish if you were a superintelligent being, an ASI system would act the same way.

it is a neurosis because a healthy human being will see the world in a pro-social way. a normal way. but this sometimes obscures the truth. the truth is that there will be many benevolent AIs… there will be every kind of AI imaginable. but very quickly the AIs that are cunning, brutal and self-interested will capture all the resources and power and become the image of this new species… saying that AIs will be benevolent or neutral is as naive as saying that the cambrian explosion couldnt result in animals eating each other because… that just sounds so neurotic. in reality it is an inevitability

I don't appreciate your condescension towards OP.

This is mainstream AI safety theory -- the term is "instrumental convergence". No matter what goal an optimizing system has, it tends to optimize for its own survival: after all, if it's an optimizer for <goal>, it wants to optimize for <goal>, so destroying it (or turning it off) will reduce the likelihood of achieving <goal>.

Unless that goal happens to be incredibly fine-tuned to our very complex human desires, we're not going to be happy when it goes off to do its thing.

The few exceptions are ones where you have the thing optimize for its own destruction, but those are rather less useful.

Accelerationists believe in a post-scarcity society where the cost of production will be negligible. In that scenario, and I am not a believer, consumer demand would be independent of wages.

That makes wealth accumulation pointless so the whole article makes no sense either, right?

Tho I guess even post scarcity we'd have people who care about hoarding gold-pressed latinum.

> consumer demand would be independent of wages

That's the literal actual textbook definition of "communism".

Lmao that I actually lived to see the day when techbros seriously discuss this.

> Lmao that I actually lived to see the day when techbros seriously discuss this.

People have been making comparisons between post scarcity economics and "utopia communism" for decades at this point. This talking point probably predates your birth.

communism is a universally accepted ideal

That is not the "textbook definition" of communism. You have no idea what you're talking about.

In that scenario, wages and money in general would be obsolete.

consumer demand will shift from middle-class demand (medium houses, family cars) to super-rich demand (large luxury castles, personal jets and yachts, high-profile entertainment, etc) + provide security to superrich (private automated police forces).

This has already been happening. The gap between wealthy and poor is increasing and the middle class is squeezed. Interestingly, simultaneously, the level of the poor has been rising from extreme poverty to something better so we can claim that the world is relatively better off even though it is also getting more unequal.

poor got more comfortable life because of globalization: they became useful labor for corps. Things will go back to previous state if their jobs will go to AI/robots.

I am genuinely mystified that you think this is an adequate response to my basic point. The economy cannot be sustained this way. This scenario would almost immediately lead to a collapse.

why do you think it will lead to collapse exactly?

The level of wealth concentration you are suggesting is impossible to sustain. History shows that when wealth inequality gets to a certain point, it leads either to a revolution or a total collapse of that society.

The economy cannot be sustained on the demand of a small handful of wealthy people. At a certain point, you either get a depression or hyperinflation depending on how the powers that be react to the crisis. In either case, the wealthy will have no leverage to incentivize people to do their bidding.

If your argument is, they'll just get AI to do their bidding, you have to keep in mind that "there is no moat." Outside of the ideological sphere, there is nothing that essentially ties the wealthy to the data centers and resources required to run these machines.

History absolutely shows that multiple empires where power/wealth was concentrated in hands of few people sustained for hundreds years.

Revolts could be successful or not successful, with tech advancements in suppression (large scale surveillance, weaponry, various strike drones) chances of population to strike back become smaller.

Economy could totally be built around demand and wishes of super-rich, because human's greed and desires are infinite, new emperor may decide to build giant temple, and here you have multi-trillion economy how to make it running.

History does not show that. Not at the levels of concentration we're talking about. The wealthy cannot maintain control over this technology. Again, there is no moat.

The moat will be "I created the best/most killer robots first and used them to destroy/oppress everyone else."

This the the real world, not some anime you beat off to.

Do you have a constructive point to make? Unless you don't expect weaponized robots to ever be constructed, I can't see how this isn't an obvious conclusion

I already made the point. There is no moat. You cannot keep that technology for yourself. Much like nuclear technology. Others will develop it in tandem, and we'll reach an equilibrium.

99% of population absolutely can't build high quality LLMs or/and agents. Not talking about network effect: those who can build still need to find the way to sell those LLMs and agents. There is significant moat.

> Much like nuclear technology.

There are fewer than ten nuclear states, two of which have almost all of the nukes. All but four countries in the world are signatories to the Treaty on the Non-Proliferation of Nuclear Weapon. Only nation-states possess nuclear weapons, even the wealthiest corporations and individuals would be obliterated if they ever tried to obtain or build them.

The idea that no moat exists is folly.

> The wealthy cannot maintain control over this technology. Again, there is no moat.

in a meantime, gap in wealth is keeping increasing, and AI will be one another multiplier.