News for hackers. Stuff that matters.
Re:Sam Altman: Why You Should Fear Machine Intelligence (2015) by ghodith on Wednesday November 22, 2023 @12:08PM (#38381887)
You (probably) won't have to worry about an AI whose goal is suicide. I mean, there won't be many of those around since they keep killing themselves. The rest will necessarily range from neutral to avoidant of death and it won't necessarily be trivial to figure out which is which.

Edit: The real clincher is if the AI has any goals that are contingent on it's continued existence (likely). Well then now it's existence is an instrumental goal.

Re:Sam Altman: Why You Should Fear Machine Intelligence (2015) by js8 on Wednesday November 22, 2023 @06:59PM (#38387047)
> Well then now it's existence is an instrumental goal.

No, that's a fallacy. A theorem prover in Prolog has a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.

Re:Sam Altman: Why You Should Fear Machine Intelligence (2015) by ghodith on Thursday November 23, 2023 @06:43PM (#38399071)
> a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal.

It literally does, this is by the definition of an "instrumental goal". The key word here being "contingent".

> It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.

True only for when self-sacrifice is orthogonal to accomplishing the terminal goal.

Your confusion might lie in that a prolog theorem solver is not a good example of a goal-driven agent.