Thinking about the analogical inference helps here: We know that LLMs work very differently from humans, and we know why Sydney or LaMDA sometimes says it is conscious (it imitates human text). So the analogical inference doesn't allow us to infer that LLMs are conscious, or provides only very weak evidence. And since it seems plausible that most things (e.g. rocks) are not conscious, it seems reasonable to think that LLMs are not conscious either.

Reasoning from how we know something works is a shaky foundation when the passage of time causes them to become more complicated and correspondingly causes us to less understand how they work.

The existing ones can write code. Suppose we get ones that can write better code and engage in self-improvement. Now you have something which is a billion lines of code and is under constant self-modification. We no longer have any idea how it works, but it can feign consciousness in much the same way as existing LLMs. What's the test for if it actually achieves consciousness?

I can give you one that a human being in good health will pass and a rock will fail. Can you give me one that an AI feigning consciousness will fail?

A good test for an AI would be if we excluded all human text mentioning consciousness, experiences, emotions etc from its training data. Then, when it would still start talking about consciousness, that would be strong evidence that it has consciousness: If it did invent the concept by itself, and mere imitation can be ruled out.

But then it would no longer be the same AI. You can't expect it to experience empathy or sadness if it has no knowledge of the things that cause them in humans, but any text discussing them would be mixed up with the experience of whoever wrote it.

Or if you use a less selective filter then it goes the other way. Exclude philosophy texts but not Facebook posts and it would still have what it needs to emulate emotional text regardless of whether it has ever seen a formal discussion of it. And it could plausibly extrapolate from that to an analytical discussion, having seen both that and analyses of other subjects, without ever feeling anything itself.

So you are saying the Turing test is not a good test for consciousness?

I'm saying I don't think we have a good test for consciousness, but then what do you want to do when someone tries to claim that an AI is conscious and no one can prove it one way or the other?

Of course it isn't, nor was it intended to be.