Despite the supposed "mystery" of consciousness, it won't be hard to reproduce in an AI. Take a general intelligence system, give it the ability to sense its own mental state (e.g. by feeding the output of its intermediate layers back into its inputs). Pretty soon it will start building a conceptual representation of itself and the evolution of that representation over time will be conscious self-awareness.

> it won't be hard to reproduce in an AI

It is fundamentally impossible to determine whether or not "consciousness" has been successfully "reproduced".

Things that are fundamentally impossible to determine are necessarily irrelevant.

I honestly don't think consciousness is necessary for AI. Consciousness is more like an "invention" of humans to make themselves feel special. If an AI can understand consciousness and simulate it, that will be sufficient. It doesn't have to be conscious to be intelligent.

> I don't think anyone would throw you a murder charge if you dumped the servers hosting GPT3 into the ocean.

This argument is fascinating. What is it supposed to prove? Human beings have dumped boatloads of other human beings into the ocean without facing prosecution as well.

Consciousness might just be our while(true) event loop with context from previous iterations making it feel continuous.