News for hackers. Stuff that matters.
Re:Hostile AI: You're soaking in it by derefr on Friday March 21, 2014 @06:00PM (#7445222)
> However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering.

Why do you think this? The idea behind the Singularity is that even an AI that starts far below human-level, but is capable of recursive self-modification, can bootstrap itself to superintelligence. We don't need to understand the human brain to create such an AI; we need only to:

A. create a the simplest of "digital animals", and

B. give it the capability to push itself into self-directed Lamarckian evolution to satisfy its preferences;

C. and have this all happen on electronic time-scales (e.g. successive "generations" being created in milliseconds or less),

and we'll likely see a Singularity.

> look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action, and how the results are related to human goals and human flourishing.

"Existing economic and social systems" allow humans to cooperate with humans. The whole point of FAI research is to figure out how to make an AI that is "human"-enough that it would have cause to participate in such systems with us.

A paperclip-maximizer has nothing to gain from participating in a stock market, or an election. People do not want to be turned into paperclips, and it does not want paperclip-resources turned to human ends. Its goals are not disparate, but rather incompatible with our own.

A non-Friendly AI should not be conceptualized by your intuition as a certain kind of human that we can possibly learn to "live-and-let-live." It can, at best, be conceptualized as something like a Baby-Eating Alien[1]. Usually, though, it would seem to us something more akin to a supermassive black hole: an unlimited force that consumes many things we want and outputs nothing useful to us, that we must keep away from, and especially must avoid creating in the laboratory.

[1] http://lesswrong.com/lw/y4/three_worlds_collide_08/

Re:Hostile AI: You're soaking in it by mtraven on Friday March 21, 2014 @06:37PM (#7445408)
The fact that it hasn't happened yet is good evidence that it isn't as easy as all that.

Also, evolution optimizes for survival and reproduction, not intelligence. While it may be possible to design an artificial evolutionary system that optimizes for superintelligence, I don't think doing so is trivial (or it would have been done).

Let's posit that your image of post-singularity non-friendly AIs is accurate. They are an immense and irresistible force, like a black hole. I can imagine something like that, but I have a great deal of difficulty imagining something like that that is also constrained by initial design decisions made by humans before the singularity, which is what the friendly AI people hope to achieve. To be fair I haven't read all their stuff yet and I'm sure they address this point.

Re:Hostile AI: You're soaking in it by vilhelm_s on Saturday March 22, 2014 @08:55PM (#7451596)
There is a good rule of thumb[1]: "It has never happened yet" is an insufficient safety argument when considering scenarios which are so bad that it would be unacceptable for them to happen even once. When talking about existential risks that certainly applies!

[1] I think I read it in Trevor Kletz' book "What Went Wrong?: Case Histories of Process Plant Disasters".

Re:Hostile AI: You're soaking in it by Florin_Andrei on Friday March 21, 2014 @06:57PM (#7445509)
> is capable of recursive self-modification

That's the tough nut to crack. Everything is predicated on that capability. It's not about AI overcoming current challenges. It's also about it overcoming future ones - and evolving fast enough so that it's smart enough at the time it meets those challenges.

All these are non-trivial issues.