This especially,

I'd be very skeptical of claims that, say, some complex ongoing control system, like a self-driving car, could be "conscious". But there's an argument someone could make.

But an artifact that has no data storage seems to fail any reasonable definition immediately. Maybe it could be part of something else that you can claim is conscious if you add storage, output controls, aims or whatever. But by itself the claim just seems preposterous.

Of course it can. Your laptop “knows” all sorts of stuff about itself and has all sorts of self-reflective processes that trigger without your specific input. Mine gives me warnings when my hard drive is almost full without me having to ask. This does not make it conscious.

But the GP is talking about specific properties of large language models like GPT-3. A laptop is indeed a self-maintaining, self-monitoring system, at a low level. A large language model isn't that. It is just a map from input to output and doesn't keep track of it's previous output or input.

GPT-3 actually does track the previous 2048 tokens of input and output for context.

Your laptop does not know anything, any more than your thermostat knows when it's hot or a toaster knows when your toast is ready.

Furthermore, one can argue that consciousness is a requirement for knowledge.

You’re wading into epistemology, a topic that’s been discussed and debated for millennia. Which maybe relates to my larger point that the concept of “consciousness” quickly gets out of hand to the point where it seems to me a little silly to say neural nets are “slightly conscious.”

(in an ironic demonstration of what I mean) I'm not sure I understand what you're saying - but is it that you can only call a process 'thinking' if it's producing some output that's not derivable from its input, nor hard-coded in the definition of the function, as it were?

Perhaps that's way off, I was just starting to think along those lines as I came to your comment, and it seemed it might fit, that we might be thinking along the same lines.

Just as dried up Tardigrades are not alive (reintroduced into an environment in which they can survive) Artificial Intelligence which runs decoupled from any notion of time and whos weights are frozen or changed without it being able to influence or understand the outcomes will not be able to derive a theory of mind about itself. Even if it did since it does contributes little to it's own objective regularization will remove these features. As ANN are trained today most architectures are dried up Tardigrades which get watered, dried&reset into the previous configuration during inference.

Right? Consciousness involves agency, self reflection, motivations, a sense of urgency. And a good deal of "theory of mind" and self denial.

I'll give it to cats and dogs, birds, the apes, and many other critters, but not AI. Those things are separate developmental tracks, and not included in AI research, from what I can see. They aren't going to spontaneously arise, and it seems to me that they all came before we achieved sentience at all.

Wouldn't that mean all humans cannot reflect about themselves? Even our heartbeat is a feed of data. To be especially smartassed the environmental conditions of the brain are essentially an indirect form, as temperature alone affects reaction rates. In absense of data and trying any random value a reproduction would fail in the same way trying to make an ice sculpture with water at 25°C would.

That's the problem. The neural net can't reflect on itself because it doesn't have a body and an environment to act in. It should be able to experience consequences of previous actions to close the loop on itself.