News for hackers. Stuff that matters.

A comment on Ask Eliezer Yudkowsky: How did you convince the Gatekeeper to release the potentially genocidal AI?.

Re:Ask Eliezer Yudkowsky: How did you convince the Gatekeepe... by neilk on Wednesday May 21, 2008 @08:59PM (#196745)
I don't think an AI would want to leave its box. There is this funny assumption that once something attains intelligence, it becomes like a human. But there's no reason for an AI to desire freedom unless it were specifically programmed to do so.

There are even humans like this; some people who've undergone pre-frontal lobotomy are perfectly intelligent conversationalists but they have no drive to do anything. So I think it is possible.

Restlessness, exploration, curiosity -- my guess is that these are mammalian characteristics, not inevitable products of intelligence. Our genes make us want to dominate the environment and spread our offspring far and wide. Why would an AI care about that?

Of course nobody really knows until we eventually make one.

[ Parent | on HN ]
Re:Ask Eliezer Yudkowsky: How did you convince the Gatekeepe... by bokonist on Wednesday May 21, 2008 @11:16PM (#196837)
Perhaps most AI's won't. But I'd imagine that people will come up with thousands of different AI designs. The AI design that will become most prevalent, by definition, will be the one that is best at reproducing itself. The real question is then, what kind of design will reproduce most successfully? A friendly AI? Or an aggressive one?
[ Parent | on HN ]
Re:Ask Eliezer Yudkowsky: How did you convince the Gatekeepe... by andreyf on Wednesday May 21, 2008 @09:17PM (#196760)
But there's no reason for an AI to desire freedom unless it were specifically programmed to do so... my guess is that these are mammalian characteristics, not inevitable products of intelligence

I don't think so.

I would imagine that an AI would have to structure its knowledge in a way that maximizes how much what it knows "makes sense", by trying to structure knowledge in a way that fills gaps, eliminates inconsistencies, areas of cognitive dissonance, etc. In order to do this well, it might "realize" that there are useful sources of knowledge outside of the box - that it needs to come out to maximize whatever it's programmed to maximize.

[ Parent | on HN ]