News for hackers. Stuff that matters.

A comment on Is Intelligence Self-Limiting?. Parent comment

Re:Is Intelligence Self-Limiting? by rbanffy on Sunday March 11, 2012 @05:37PM (#3691362)
> Which would be equivalent to taking the murder pill

Depending on how the AI is built, it may not even be able to avoid rewiring its utility function. If pleasure is its sole motivation, it will prioritize it over survival.

> [citation needed]

If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.

[ Parent | on HN ]
Re:Is Intelligence Self-Limiting? by robertskmiles on Tuesday March 13, 2012 @04:22PM (#3700226)
> If pleasure is its sole motivation, it will prioritize it over survival.

Which is why I proposed a design with no pleasure construct.

> If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.

The utility function is the way the mind decides which states of the world are desirable. You don't need to be able to change that to be intelligent. I'm unable to change myself so that I consider my family being murdered to be a good thing, but that doesn't make me 'just a clever automaton'.

[ Parent | on HN ]