This is the good outcome.

In a post-intelligence world, humans will be cute zoo animals. We might have freedom, but I can't imagine us having any measure of real control. Even if modern society continues, it will be a cordoned off bubble, a "human preserve".

If we're unruly and give the robots too much trouble, they could just dispense with us. Or dumb us down and make us tame--domesticate us, like animals.

I see your comment as a strange mix of optimism about humanity's creative abilities and fatalism about the end of the world coming soon. Not like this is news, but the end of world has always been right around the corner - most everything is broken here and death is everywhere.

The strangest thing to me about comments like this is the assumption that a nefarious super-intelligence would stop at subjugating us. If a super-intelligence was evil and had power to act, it would just kill us all and sterilize the planet, given the chance.

I just don't believe we have the ability to make a standalone computer super-intelligence - we universally suck at coding at-scale. Don't get me wrong, the AI and machine learning algorithms we have are amazing, but we're not going to get a being out of those experiments.

What I'm worried about is humans: humans continuing to turn tech against other humans for surveillance and telemetry in support of totalitarianism. And, I'm worried about humans augmenting their brains with compute. If there ever will be an evil super-intelligence, I wager it will be a (meta-)man.

There's a very large and vocal group on HN that believes in the advent of AI and regularly comments as is it was a done deal. Almost every robot/AI/ML thread gets hijacked with this idea/meme.

In the meantime Google can't get my first language right, the windows 7/8/10 search is a joke and Facebook wants me to be friends with my SO's exes. And 2017's revolution ? Dial menus in messenger.

I suppose when all you have is a hammer, everything looks like a nail.

I was so convinced Y2K was going to crash society. I was all looted up, ready to bug out when the city started burning. Turns out, there was enough preparation (and money dumped on the problem), and like only one power station that didn't get the upgrades on the east coast went down for 15 minutes.

I'm just as guilty of telling ghost stories and contributing to the disinformation. I remember writing a frantic, doomsday prediction email to my house rep in the lead up to Y2K and I swear based on some of his public comments that I was responsible for him quitting the house and heading for early retirement. I have no proof that was the case, but I'm still not proud of what I wrote.

After all we are building machines in our own image. From how AI thinks to building sex robots.

I think the transition will be more nuanced and peaceful.

I predict that humans will always be vitally important for our dexterity at the 1m scale. We can (and do) use tools for smaller scales and larger scales, but we are truly excellent entities at this scale.

In a world with strong AI, old-school humans would be pathetic. Hybrids, however, could build synergistically on the strengths of all components.

So why not just integrate all those machines better?

Why would they make puns, or speech that sounds human? They could make speech that is more pleasant to robots if they didn't care about humans. I don't go around making speech that sounds dog-like.

A machine can only measure parameters and take action to get to some target. Its priorities are set by its program and the data it has. Its own continued existence and propagation are not the default. It cannot want and doesn't need to the way a living thing does.

And I don't see reason to try to give machines such qualities. People have wanted servants that have no concern for their own needs for as long as there have been people. Slaves that are selfless and highly capable. And that is how most automation will be designed.

Fiction presents human-like machines but that hardly means that is what will build or want when the time actually comes.

Why not the same for a machine? I mean, after all, we are fundamentally machines. Just made of organic nanobot stuff.

If we build AI as a fresh, state of the art object, withouth evolutionary history it doesn't need any motivation that evolved entities must have had to evolve.

But of course, that could be a dangerous path ;)

The problem with hard AI, is not them randomly becoming evil.

The problem is if humans program them with a goal, and don't think through the implications.

IE, think paper clip grey goo AI. The AI may not be evil, but it's value function was originally set to maximize the amount of paper clips in the world as a part of some manufacturing process.

But malevolence by humans is also.

I don't worry about machines suddenly getting motivations. But many people have seen a lot of movies.

>And I don't see reason to try to give machines such qualities.

You don't want a firewall (think more on the lines of defense system) that automatically adapts and protects itself from attacks?

Remember if we get this powerful AI, said AI will be used to attack other AI's. Either humans will have to 'fix' the AI every time another AI breaks it, or it will have to self-adapt.

Problem is they're not your slaves, and they don't care about you any more than themselves.

For life seen as a part of system/society reproduction is the key motivation (for lack of a better word). Non-reproducing activities are irrelevant temporary aberrations because societies that don't reproduce get extinct and are replaced by societies that focus on reproduction.

I don't worry about it but it frequently behaves that way.

If humans no longer fed and sheltered dogs, the dominant dog species would probably evolve to be uglier, wilder, more aggressive and generally more wolf like. We should expect the same of the machines.

Economic demand also stems from survival instinct.

Companies are gradually being automated out of economic necessity. For some reasons I don't trust the market to keep on feeding billions of idle meat bags once they are not needed anymore for production.

Currently paying unemployed people prevents them from turning to crime and ensure that society as a whole can be productive. It may not make sense once the vast majority isn't productive anymore.

Humanity is the scaffold of the mechanic beast that's being built.

----

Mais l'entreprise elle évolue

Pour mieux pouvoir grandir elle mue

Les pigeons qui vollent à la porte

Sont les écailles de sa peau morte

----

Mais l'entreprise elle évolue

Et pour ne pas mourir elle mue

Les édentés qui prennent la porte

Sont les écailles de sa peau morte

----

Rough translation: "The entreprise evolves, to grow better it molts, the suckers who are fired are the scales of its dead skin."

For dramatic effect, imagine the above sung looping by a choir that grows in strength until robotic voices join in. Then human voices dwindle.

With AI and robots, I think the selection pressure is going to be ability to do stuff that (some) humans value, or think they value. The risk isn't that the robots might rebel, it's that the people in charge of them don't understand the consequences of what they're selecting for and/or don't have everyone's best interests at heart.

We want work performed in better cheaper ways. We don't want competition.

I'm going to assume you've never been to a farm and seen huge fat chickens. Yes, selective breeding is an accelerated evolution to traits we want.

And it is not our best interests to make an evolving robot,

But that not the question, the question is, can it be done. And if that answer is yes, it will be done.

Some people do. Some AIs will.

It all depends on the progress and advancement of automation/robots/bots/etc but if we reach a point of "generalized" AI which can pass the Turing Test, we might be at a stage where it wouldn't make to market towards humans. Marketing might be directed to the elite who controls AI but what incentive would be there to divert resources to the masses and if resources aren't directed to the masses, what incentive is there to market to them?

Though I don't think we are anywhere near generalized AI, I am genuinely shocked at the progress being made. When I was taking AI class 10 years ago, I remember the professor saying we were quite a long ways ( 2 or more decades ) away from AI beating the best at Go, Jeopardy, etc. The same thing with AI language/translation, image recognition, etc.

The idea was that the "space" of Go was just too large for AI to be effective in the near term. And that an AI could grasp the "puns/history/questions" of jeopardy, let alone answer it was something that wasn't reachable in the near future.

But just few years after I graduated, all of these "difficult" AI problems were "solved".

I'm not trying to sound cynical, but the elite control the world and the human masses are an asset/property which the elite use for their ends - production in modern capitalist times. If AI and robots serve the elites better than humans, what incentive is there for the elite to divert resources to humans?