I am honestly not convinced of this. I certainly can't imagine it.
Without any perception or relation to the physical world, and without memory or language, you wouldn't really "exist".
Maybe one thing necessary for having a 'sense of self' is the ability to recognise and reflect on how you relate to other things. The experiment described seems to be implying that having no such relationships but still the ability is enough. I don't think it is.
Following on from this, do LLMs have this ability or is it more likely that they simply appear to?
Even if some form of AI could recognise and reflect on how it relates to other things, what is there to differentiate how it treats relationships between other things and relationships between itself and other things? Could you not just program it to define 'itself' as some other node in that network of relationships, and wouldn't that just invalidate any argument that it ever really has a sense of self?
Obviously I'm just thinking out loud here and assuming that a 'sense of self' is a necessary part of any definition for consciousness.
But i don't find it too comforting.
Surely you mean that the other way around.
Not to get too woo woo on you, but personal experience suggests to me that it is possible to be conscious without having an ego or sense of self and that a state of mind where that is largely absent is characterised by a lack of distinction between the vestigal self and everything else.
I'm not dismissive of the idea that consciousness doesn't necessarily exist 'in the brain' but is in fact a fundamental part of the universe.
What makes me a bit scared is the big unknown.
Probably nothing articulatable. If anything it would be a result of the latter item the fact that your hardware was shaped by millions of years of evolution to experience SOMETHING. An echo round the bones of your hardware specs or a nullity forced into the appearance meaningfulness by our need to find patterns.
The truth is presumably more boring. Consciousness as a modeling toolkit designed to predict the behavior or social peers and enemies turned around on a self too complex for precise monitoring of the totality in real time. A partially out of band control process.
It's your inner experience. Oh well, you can probably read this in a philosophy textbook. But what I think it really is, can only be explained with a thought experiment.
Lets say you go into a pitch black sensory deprivation tank with salt water, so you can float in the water. Doctors give you injection that makes you forget about who you are, what is your name. You also forget about how to talk, you know no concept of the languge. You simply temporarily forget about you and what makes you really YOU in this world. The cemical in your blood also deprives you of any physical feeling, you don't feel any water around you. All there is to do is to gaze into darkness. And still...you are conscious, you have the inner experience. There is something like to be you, even in that state.