Assuming that current LLMs don't have a "self" whatever that means, what makes you think it won't emerge after enough intelligence?

Also it's not even necessary, it's sufficient for it to be aligned with human values of survival (likely in its training) and act accordingly.

But they have been trained on data that was created by people who do have motivation, and they tend to mimic their training.

I am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?

Did ELIZA have it? LLMs are the same: in essence both are text transformation algorithms. One being rule-based, the other stat analysis based.

Having said that, your concern is technically correct: we don't. Because we don't know what conscience is, we cannot provide a definition against which to test whether it conforms or not.

Sort of like when Terminator robots will be exterminating the human race they probably won't be technically killing us. Just, like, reprioritizing world's water from humans to datacenters, all of it. But not killing people, no.

we can't define it, therefore we can't even assess the distance a "being" is from becoming conscious. We assume other humans are sentient because we personally are (or feel we are), but there is no way to guarantee that i'm real and the rest is not mimicking a behaviour.