News for hackers. Stuff that matters.
Re:What Darwin's Theory of Evolution Reveals About Artificia... by _delirium on Friday June 22, 2012 @12:09PM (#4147373)
That's definitely true, and a major problem beyond friendliness (i.e. way before any singularity). A traditional assumption in machine learning and other areas of AI is that you can factor out the problem-specification parts from the algorithmic problem-solving parts, but it's very easy to get results that are unexpected, except in the rare cases where you have an 100%-correct obvious objective function handed to you by the structure of a problem. In cases where a human is writing down what they think the objective is, a very common case is that there are a lot of unspoken things that they intended to be included, "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do". Hence a lot of iteration on objective functions is needed in real-world applications, and recent-ish research focuses on alternative formulations like interactive machine learning, preference elicitation, etc.
Re:What Darwin's Theory of Evolution Reveals About Artificia... by adrianN on Friday June 22, 2012 @04:58PM (#4148765)
For an example of this have a look at this essay [1] by George Danzig in which he tells the story of how he tried optimizing his diet using linear programming.

[1] http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf

Re:What Darwin's Theory of Evolution Reveals About Artificia... by goostavos on Saturday June 23, 2012 @01:01AM (#4150094)
Interesting read. Thanks a lot!
Re:What Darwin's Theory of Evolution Reveals About Artificia... by endtime on Friday June 22, 2012 @07:08PM (#4149354)
> "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do"

This is similar to the basic idea of Yudkowsky et al's Coherent Extrapolated Volition, the gist of which is that an FAI's metaethical function should be some aggregated of what it believes people would want it to do if they had the knowledge and intelligence(/cycles) that the AI has. (I believe there exist updates/improvements to CEV, but I don't know much about them.)