That said, things change once you get a body. If you put ChatGPT into a simulated body in a simulated world, and allowed it to move and act, perhaps giving it a motivation, then the combination of ChatGPT and the state of that body, would become something very close to a "self", that might even qualify for personhood. It is scary, by the way, that such a weighty decision be left to us, mere mortals. It seems to me that we should err on the side of granting too much personhood rather than too little, since the cost of treating an object like a person is far less than treating a person like an object.
If the entity doing the cognition didn’t evolve said cognition to navigate a threat-filled world in a vulnerable body, then I have no reason at all to suspect that its experience is anything like my own.
edit: JavaJosh fleshed this idea out a bit more. I’m not sure if putting ChatGPT into a body would help, but my intuitive sympathies in this field are in the direction of embodied cognition [1], to be sure.
The only reason our brains exist at all is to control our bodies. There are no conscious entities we know of that are just brains in vats.
Why would a being that has no need to maintain biological homeostasis experience something like hunger, restlessness or tiredness?
That’s what I mean when I say I’ve no reason to believe a thinking machine that has none of my motivations and limitations would ever have a “subjective experience” like my own.
That is, of course, unless we succeed in reverse-engineering human consciousness and somehow deliberately program the machine to experience it.
Additionally, it's not hard to imagine it could recognize that the acquisition of fresh training data would be an ongoing survival need.
Machines developing agency is the basis of lots of dystopian terminator-style sci-fi.
To be clear, I don’t doubt that it might be possible for an engineered thinking machine to develop a subjective self-experience.
It just seems clear to me that, if such a scenario arose, I would have absolutely no idea what that would be like for the machine. Its needs and circumstances (and, therefore, its behaviors) would be utterly different from my own in many non-trivial ways.
Given this, I don’t believe it logically follows that a machine that attains self-awareness will necessarily develop something as singularly “human” as the desire for vengeance, for instance.
An AI model needs to evolve its cognition to successfully navigate its training environment for survival.
Shouldn't that satisfy your criteria for suspecting its experiences to be like your own?
The AI models we have now will happily run forever if supplied with external power.
That’s quite different from the average human being’s experience! Our biology imposes so many limitations on us that computers don’t have to deal with, and many of these are cognitively non-negligible.
If an AI model ever emerges that can get bored— or tired, or lonely, or annoyed, or anxious, or scared— then I would begin to take seriously the possibility that it experiences existence something like the way we do.
A human will run forever as well if supplied food (external power), a new body (computers degrade) and genetic data protection (hard drives corrupt) :)
While AI models may not be quite there yet, their world seems fairly similar to ours in the important ways of environmental constraints and survival pressures