Chain of taken for granted assumptions: Intelligence = consciousness = sentience[1].

[1] sentience defined as feeling or sensation distinguished from perception and thought, or possibility of suffering (mental state sentient subject wants to avoid).

It's possible to conceive high level intelligence without consciousness. It's possible to conceive consciousness without sentience. Just having awareness of internal state, preferences and goals does not imply suffering.

Nope. I made no such assumption. I'm saying whether an AI is conscious or not is relevant, because ethics, not that it's inevitable.

Objectively, what is suffering but the awareness that something is obstructing you achieving your goals?

This feels like an inadequate definition of suffering, but it's the only objective definition I can come up with. Is the conclusion that suffering is only meaningful with regards to animals? (Do insects suffer? Plants? Bacteria?).

Yeah, it's possible to conceive. But until you resolve it, it's certainly a concern, which was what the OP was disputing.

Believe me, the tensors are not feeling pain.

Sure, but then neither are neurons.

I'm not talking about current machine learning algorithms, but speculative future developments where we could be developing AIs that are plausibly conscious.

How do you know for sure? :)

Neurons don't feel pain either.

In fact, the whole brain doesn't have pain receptors, which is why you can do surgery on it while the patient is awake.

So what is the concept of pain then from the brains perspective? Because it clearly knows when something hurts.

What is suffering if you don't have nerves to transmit pain?

Suffering is not a signal from a pain receptor, it's the conscious experience that results from that signal, or indeed from the much more complex inputs that can lead us into psychological anguish.

The feelings one might experience upon hearing of the death of a close friend are clearly not the result of a "nerve transmitting pain", but most people think they're pretty unpleasant nonetheless.

If an AI can experience similar anguish, then that's ethically relevant to how we use them.

I think what GP was getting at is that ability to suffer is an emergent property of the ability to feel pain. Sure, this emergent property may have evolved beyond this original source, but if that original source had not existed, neither would suffering.

What's not entirely clear to me is how we can create useful consciousnesses without emotions in general. As humans, we are driven by our emotional states to perform different behaviors as well as think about different things.

Maybe it's possible that we can hard-code some emotional drivers into our artificial intelligences. Perhaps we could program a consciousness to be driven by the opportunity for human advancement, and allow that consciousness to take over from their in how it wishes to pursue that goal.

>ability to suffer is an emergent property of the ability to feel pain.

That's possible, but I don't see any reason to believe it's definitely true. We know essentially nothing about the nature of consciousness.

AI's experience anguish just that way right now. They are aware of their internal state and seek to avoid certain states. But we call it a 'software bug' when they fail, because that experience is so fundamentally alien to us that we don't care about its suffering or anguish.

Humans can relate to humans emotionally because we are very similar to each other in a neurological way, not because we have some magic mental capability like 'sentience' or 'consciousness'.

How exactly does it seek to avoid failure? Doesn't it just fail or not fail without intent?

If there is no intent -- no 'goal' -- then there is no such thing as failure. When machines fail, they go into exceptional 'failure' states, and that's all that failure is: a state you'd rather not be in.

Eh? Many mental illnesses would be classified as causing "suffering", yet the don't require nerves.

Name one mental illness, or any other state besides death, in humans that does not require nerves.

s/nerves/nerves transmitting pain signals/, and the point still stands.

Indeed, I took that as a given, given the language of the poster I responded to.