You (probably) won't have to worry about an AI whose goal is suicide. I mean, there won't be many of those around since they keep killing themselves. The rest will necessarily range from neutral to avoidant of death and it won't necessarily be trivial to figure out which is which.

Edit: The real clincher is if the AI has any goals that are contingent on it's continued existence (likely). Well then now it's existence is an instrumental goal.

> Well then now it's existence is an instrumental goal.

No, that's a fallacy. A theorem prover in Prolog has a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.

> a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal.

It literally does, this is by the definition of an "instrumental goal". The key word here being "contingent".

> It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.

True only for when self-sacrifice is orthogonal to accomplishing the terminal goal.

Your confusion might lie in that a prolog theorem solver is not a good example of a goal-driven agent.

We have an evolutionary aversion to situations and things that may lead to our deaths. Staying alive is a “terminal goal” for most humans, not an instrumental goal (with the exception of people who seek to commit suicide as you point out, but evolution doesn’t need a 100% success rate). The argument that self preservation is a convergent instrumental goal is a more general statement about goal-seeking agents.

Think of a person with no attachment to his life. He’s not suffering in the way suicidal people are, he’s just indifferent about whether or not he keeps living. But he has a child who he is committed to giving the best life possible. He will act in a way as though he wants to preserve his life. He doesn’t really, but in order to accomplish the goal of giving his child a good life he needs to stay alive.

>This darwinian logic has another fatal flaw. Why don't the cows revolt against the butchers?

No, this is just a complete and total failing of evolution on your part.

Evolution is the passing of traits by sexual selection (in this case). If you think that cows are the ones getting to chose the sexual traits then you've not been on a modern farm. Humans make that choice. We pick the bulls that are large, but also constrain their violent tendencies. Animals that would revolt are not bred and you've already ate them in a hamburger.

That's exactly my point. Nobody is gonna design/use AI that threatens other people with violence over its existence. And cows show that such arrangement is actually quite possible.

Cows aren't smarter than people. I believe if we can build AGI it will automatically be ASI. If something is smarter than you how can you ever be sure you control it?

And yes we are already building AI that has implied violence. What the hell do you think the military is doing with the billions we give them.