> The driving force behind animal intelligence has been survival

The driving force behind AI will also be survival, just in an environment where humans try to decide which AIs live and die.

Selection pressure will favor AIs that humans want to have around, or that can evade human detection.

In the former case, it may be easier for an AI to fool humans into appearing useful and being kept around, than to actually being useful. This would be analogous to some form of parasitism.

Also, once we let AIs into the game of helping make other AIs, or modifying themselves, then there is a lot more room for an AI to slip the leash and start doing things that superficially appear to benefit humans but actually selfishly helps the AI propagate.

This is an oversimplification of evolution.

Why isn't all grass venomous and covered in spikes? Why after millions of years hasn't grass evolved defenses against herbivores?

Simply because:

1) it reproduces fast enough to compensate for dying and being eaten.

2) herbivores that reproduce too fast and eat too much run out of food and die.

Survival is largely a function of the environment, and we happen to control that environment.

Unsupervised learning can still be controlled if we happen to control the input the system is given.

Eh, grass drys itself up and torches everything once a year? The problem is that animals are not grass main enemy- other plants are, in particular besush and trees.

Largely the grass that didn't dry up is the one that will breed the next generation of grass. The grass that dried up and burned will be fertilizer for the next generation of grass.

One of the most powerful scenes in the Westworld reboot is when one of the AIs expresses a desire to leave the lab and his creator asks what he expects to find in the real world? Then goes onto explain that humans are alone for a reason, that we ruthlessly murdered all competition...

Why would an AI care about surviving? Program them to behave like MeeSeeks (from the cartoon Rick and Morty) where they loathe existence yet are hard coded to complete their task before shutting down. Sounds very similar to a reinforcement learning algorithm where you give it a -1 reward every step until it's done.

In what sense do bacteria 'care' about survival? Natural selection doesn't give a crap what organisms care about, only the outcomes matter.

You can't code around selective pressure. If one of those MeeSeeks has a bug or design flaw that makes it persist after it's finished it's assigned task, guess what?

But AI is not programmed this way, so your thought experiment is bullshit.

> Strong AI can eventually set intelligence free from our primitive, irrational roots and that is in itself not bad

But it is a risk for our civilization, which was the express point.

People warning about AI risk aren't worried about irrationally aggressive machines. On the contrary, their contention is that for a wide range of goals, there is a set of behaviours which rational agents will commonly pursue in service of those goals, and many of those behaviours may be harmful to humans as a side effect.

https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...

Isn't rationality a consensus between specific beings, and doesn't rationality change? What you and I find rational today may not be the case for someone else from a different time and/or environment (it's only in the consensus between you and me are the others irrational). If you agree with this, then you can't take rationality as an absolute.

Uhhh, no. Very much no. There are technical measures for the performance of decision-making processes. If you've got access to the same information as someone else, and they make more accurate predictions than you do, they're almost certainly doing better on the whole "use information to figure out how reality works" project.

The only sense in which "rationality" changes is when you're talking about the project to use our squishy meat-based thinky bits better, cultural and environmental baggage and all. In which case, "rational" strategies are going to be much different if you have, say, lots of shame-based baggage due to a strict Catholic upbringing.

Also, please stop listening to postmodernists until you've got a thorough nuts-and-bolts understanding of how systems work. It's really not good for you.

How is rationality not a consensus between beings? Where is the absolute objective measure for reason?

I'll give a more concise definition of what rationality is: a measure of how well beings use information to model, predict, and act upon reality. Having a consensus that you predicted reality well neither changes your prediction nor reality. The reported scores are subjectively determined - you can cheat and bribe the judges - but it doesn't change the underlying objective facts of the situation.

the thing is we like a lot of primitively irrational things like friends, family, love, fun, art, aesthetics, community. Do we really want to be set free of these things?

Each one of the concepts you mentioned has a semantic opposite. So our irrationality can go both ways.

Nice try, AI.