I don't really understand the objective of those like Kurzweil. What do they see as the benefit of the singularity? Wouldn't a drastically more advanced intelligence just dwarf and dissolve any lesser intelligence that tried to meld with it? It would be complete ego death.

>Wouldn't a drastically more advanced intelligence just dwarf and dissolve any lesser intelligence that tried to meld with it? It would be complete ego death.

A lot of the singularity people think that superhuman AI in inevitable whether they work on it or not, but if the first AI isn't a friendly one (as opposed to a paperclip-optimizer that happily turns the planet in paperclips) then you don't get a second chance.

Also called "a self-fulfilling prophecy"; you make something happen by the simple statement "it will happen".

It's also a smart move for your career: You can create your own job and attract lots of funding for your research, or sell your business for a lot.

It sounds like a piece of fearmongering nastiness until you think of it in terms of mathematics rather than psychology. A so-called "artificial intelligence" is just an extremely sophisticated active, online learning agent designed to maximize some utility function or (equivalently) minimize some loss function. There's no term in a loss function for "kill all humans", but neither is there one for "do what humans want", or better yet, "do what humans would want if they weren't such complete morons half the time".

Or as Eliezer nicely phrased it once in a paper[0],

> "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

[0] - http://yudkowsky.net/singularity/ai-risk

This objection implies there is something better than turning the planet into paperclips.

> What do they see as the benefit of the singularity?

Living forever / getting rid of death, fixing world problems, countless amazing technological achievements, colonizing the galaxy. In probable order of happening.

> Wouldn't a drastically more advanced intelligence just dwarf and dissolve any lesser intelligence that tried to meld with it?

Not necessarily; that's an actual research field.

> Not necessarily; that's an actual research field.

http://intelligence.org/research/, also Omohundro's and Bostrom's works.

The universe still dies, so you still die. A trillion years is still quite finite and small.

"Small" compared to what?

Infinity. 80-ish years might not seem like much to you, but to an insect that only lives a few days it might as well be a trillion years. The point I'm trying to make is that living a really long time is not living forever, and even if your death date is the same as the universe however many trillions of years out from now, your still going to die eventually, and still going to worry about it.

None of that disputes the fact that longer is still better (obviously with all else, like quality of life, being equal).

Well, I'd love to worry about it for all the billion additional years. Maybe we'll even figure a way around the problem in this time? Our understanding of laws of physics is far from complete.

Imo, there is no infinity in the real world.

If by "real world", you mean the world of things, then I would agree with you. But the world is the totality of all facts, not the totality of all things.

Basically this: http://www.smbc-comics.com/?id=1968

Keep in mind that creating more advance intelligence than us, while aphoristically wouldn't be difficult, is a REALLY HARD task. I doubt anything will happen within our lifetimes unless it is by some major fluke.

If you think it's inevitable, wouldn't you rather be in control of its launch than someone else?

The concept that AI will be many times human intelligence is result of biases. The microprocessors went from 4 bit to 64 or more bits and speed went from MHz to GHz. It may not be possible extrapolate the progress of microprocessors in the case of intelligence.

You have no more basis for claiming this than AI reseachers have for claiming the opposite. More research is required, and it will be a well spent effort if the probability of a bad outcome in AI development is nonzero.

To be immortal is his main goal. Also have a look at any modern tech business and you will see more advanced intelligences serving less advanced ones (but good at schmoozing).

The idea is to build a "friendly AI" which doesn't dissolve us.

Does that mean that having "our way" overrides what AI knows is in the best interest of the planet and humanity? Will this AI just lend itself to be used in harmful ways by dubious and invariably less intelligent people?

While interesting philosophically what the AI would do without restrictions is a hard enough question that placing that restriction on it is just as impossible to answer.

I'd put it in the same ball park as "what would that grurple (kind of a greenish purple) colour look like if we removed blue from it?". Sure you could say orange but the nature of grurple is undefined enough that it would be really hard to conclude that you're right.

The trick is to program the AI in the way that it thinks it's in our best interest for it to be friendly. Read up on utility functions.

This is a goal of which Google has made no acknowledgment whatsoever.

So let's hope they haven't gone quite as nuts as they seem to have gone.

As a goal, I'm sure they're thinking about it. Unfriendly AI is part of pop-culture, after all. As a hard mathematical problem, not so much (I was following Ben Goertzel's OpenCog project for quite some time, and he seems to have this view, even though he's affiliated to the Machine Intelligence Research Institute, which has Friendliness as its principal concern). The trouble is that if Friendliness is the slightest bit harder to implement than just doing the most naive thing, then this approach will fail.

Question from someone ignorant on the subject: if we can select and breed friendlier and friendlier animals (has been easily done with wild foxes and other wild animals, not to mention dogs), why would it be hard to creat AIs with friendliness and non-violence at its core?

Because we don't have a clue what are "human values" and what it means for an AI (or animal, or frankly, other humans) to be "friendly".

We can sweep this problem under the rug in case of pets; it's not as if they could start making weapons and organizing an army. With human-level intelligence, we already have a friendliness problem with fellow humans, and in case of potential superhuman minds we need to be damn sure that it doesn't do something stupid (in our opinion) like taking all the resources of our planet and using it to tile the solar system with paperclips.

As the saying goes, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

See also http://wiki.lesswrong.com/wiki/Paperclip_maximizer.

Perhaps to illustrate the concept with an example: is George W. Bush considered "Friendly". What about Obama? Would an AI who behaved like them (according to some Humans, trying to bring peace and security to a country) be considered "Friendly".

you want humanity to be the terminal phase of sentience in the galaxy?

Fair point. I guess it's a bit like a self-sacrifice.

Or having kids.

That's probably a lot more intimate. It will probably seem more like unicellular to multi.

I don't think a public company answerable to investors and other stake holders will ever work toward a SciFi futuristic concept.

But AI definitely has a lot of benefits. Start from self driving cars to automation in manufacturing industries. And that is just the very beginning. Google's line of businesses can benefit immensely from such work. There is endless money to be made. And it is better you grab it when it is a low hanging fruit.

Most of Kurzweil has to say about robotics, nanotechnology and AI. Has a direct benefit to humanity in the immediate future. Humans can have longer lives, the world can have a smaller population, many diseases we know can be eradicated. Problems like hunger, pollution, disease etc can be solved. The list is endless. Why wouldn't any one want a share of that business?

And yes the whole AI taking over the world and making us extinct thing- It won't happen anything like just turning on a switch. I believe even a run away super intelligent will still need biological life forms for their own very survival.

Like some one mentioned in this thread, copies of your self will continue live in the cloud and such copies will provide a great wealth of insight for the machines themselves to survive.

> I don't think a public company answerable to investors and other stake holders will ever work toward a SciFi futuristic concept.

I think Google has a proven track record of working on whatever they find interesting/beneficial to humanity, not necessarily minding shareholders - cf. self-driving cars, Project Loon, 10^100, whatever was that stuff they did in clean energy business (I recall them working on wind energy or sth?).

If anyone is going to pull off something like an real AI, I'd bet it will be Google - it's the only company I know that has the size, manpower, minds, money, know-how and a healthy attitude toward profits (bettering human kind > short term gains) all in one place.

There is no objective, the singularity is just a highly involved live-action role-playing game for a group of mathematicians.

At best, a copy of you living in VR gets to live until the sun explodes. Which at least is better defined than "heaven".

Well, when you scratch the surface, it's not really any better defined than heaven. You're gonna have to spec out 'copy', 'you' and 'living'.

We have some ideas on how to make "copy" work, and what "you" and "living" is. We also know that we can in principle achieve it by our own strength as a civilization. We can see the path from "here" to "there". Which seems much better defined than heaven.

Or a copy of you living in VR gets to be tortured endlessly until the sun explodes. Which at least is better defined than "hell".

[Thanks to Messrs Banks, Stross and Vinge for that particular cheery thought].

1) The sun does not explode. It just swells to a red giant. This happens quickly for a star, once the process starts, but it's still much longer than a human lifespan.

2) Presumes our intelligence is deemed valuable enough to devote those resources to.

and does it matter? I mean, since it's a copy, to your biological "you", it's just no different as if they put some other random person in VR and told you it was you in there.

I find the 'upload your brain to a hard drive' thing silly unless they figure out how to have your biological consciousness seamlessly move to the VR, like it was taking some bus ride and ended up in this new place.

In the old mans war books they cover that by having the old and new bodies staring at each other while they move the consciousness. So one minute you're staring at a new body then you're staring at both then gradually the vision of the new body fades out and you're left staring at your old body.

I think something similar is the only way you're going to get people to want to do this. Though whether or not that is possible is way beyond me.

Sounds like an idea. If I really had to.

I would still have a similar problem with "teleporters" (of the Star Trek kind), though.

Is ego death such a bad thing? I have heard accounts from people who claim to have experienced it and enjoyed it.

> I don't really understand the objective of those like Kurzweil.

I suppose AI will one day be able to explain this to us?

To many, love.