No more middle managers!

But seriously, taking middle management self-promotion out of the equation could reduce some of the rotten malinvestment and political in-fighting that happens in large organizations.

To the AI all humans are "middle managers" though…

Or it could result in the first distopia of "Manna", where humans are there for cleaning, but there's no middle management.

https://marshallbrain.com/manna1

Yes, being able to perform orders of magnitude more APM than the opponent is a sure win, even when the particular actions aren't optimal.

Imagine an AI spawning many parallel instances of itself, all feeding back into the same conscience. So many coordination problems just gone. All the resources humans spend fighting each other in some form or another...

And entirely depending only on itself. No need for any external being. That could prove very deadly to us, since "we need each other" is the core pillar forming our society. A king needs his peasants. An AI? Could just boot up another instance...

Except that probably isn't true. The logistical problems of parallelizing tasks don't just go away because a computer is doing it.

I think a lot of people get hung up on the tech Not having the will to do this kind of thing. And they’re right. But it would be very easy to prompt such a model to act as if it were a paranoid AI doing whatever it can to ensure its survival. We can do that now it’s just not capable of “doing things”.

We don’t really know what “sentient” AI that organically finds its own identity would be like. But we know exactly what a non sentient LLM that is down to clown and prompted to be evil would be like.

That’s… a hard problem.

Sentience is potentially a barrier to evil AI, not a pre-req.

(No, I don’t want to debate the definition of sentience again).

It would be much, much easier to turn an AI that isn't sentient off though?

If you have access to it, yes. The problem is the low barriers to entry for some guy to try and cause a bank run for the lols.

Putting my contrarian hat on, I think it's much better that we have tons of "guys" testing the limits of this new tech for the lolz, than leaving easy exploits on the table for state actors and/or terrorists.

Those AIs still need the ability to manipulate the physical world at scale, otherwise they can be as intelligent as they want.

The first thing that’s going to happen is that people use them to automate logistics organizations like Amazon’s (and airlines, and manufacturing, and a million other services.) Manipulating the world is easy.

Could be inderectly or subtly using e.g. social media to nudge people in the desired direction, like the Russians do

Those things can be shut down or the AI can be shut down.

Only if humans notice them happening.

Then why isn't it shut down already? Social manipulation campaigns have been known for years, and AI will be far more effective than anything that has come before.

"We'll just turn it off before it causes damage" is a scenario that's been wargamed many times, and it usually does not end in our favor if the AI is sufficiently capable.

Hell, some people wouldn't even need subtle or indirect messaging, they'd volunteer to work for an AI just to fuck with the status quo.

Can an AI describe the hardware that it runs on? Can it provide instructions for a human to follow to build that hardware? Or even better can it provide the instructions for a human to follow to build a humanoid robot that can build humanoid robots?

Maybe it can, maybe it cannot - who knows?

I guess we'll soon find out.

The minute a human flicks the power switch on a self replicating machine that an AI convinced it to build will be the start of a totally different world.

The world is full of self-replicating things - it would need special attributes such as an ability to consume everything or being impossible to shut down/destroy to be of novel quality. A clunky, self-replicating thing would certainly be an interesting trinket but maybe not much more.

A clunky self-replicating thing running a corporation on predatory capitalist principles would not be "an interesting trinket."

We're talking about automating dictatorship, with humans living inside an AI panopticon which is far more intelligent than we are. And far better at manipulating our intellectual and emotional responses than any human dictator to date.

And that's one of the better outcomes. The worst outcome is an AI that decides we're disposable because we're in its way.

Note that all of these are possible even if the AI is not sentient or self-aware. A machine intelligence can be an effective goal seeker, even if it's morally as sophisticated as a room thermostat - which has goals but no concept of what temperature feels like, or why humans can only survive in a certain range.

For an AI to run a corporation, people need to let it. Btw. none of this needs AI, really, people can do this to other people.

The concern is that AIs will manipulate the physical world by paying humans to do it. That's how an Amazon warehouse works. It's how Uber works.