Maybe one thing necessary for having a 'sense of self' is the ability to recognise and reflect on how you relate to other things. The experiment described seems to be implying that having no such relationships but still the ability is enough. I don't think it is.
Following on from this, do LLMs have this ability or is it more likely that they simply appear to?
Even if some form of AI could recognise and reflect on how it relates to other things, what is there to differentiate how it treats relationships between other things and relationships between itself and other things? Could you not just program it to define 'itself' as some other node in that network of relationships, and wouldn't that just invalidate any argument that it ever really has a sense of self?
Obviously I'm just thinking out loud here and assuming that a 'sense of self' is a necessary part of any definition for consciousness.
But i don't find it too comforting.
Surely you mean that the other way around.
Not to get too woo woo on you, but personal experience suggests to me that it is possible to be conscious without having an ego or sense of self and that a state of mind where that is largely absent is characterised by a lack of distinction between the vestigal self and everything else.
I'm not dismissive of the idea that consciousness doesn't necessarily exist 'in the brain' but is in fact a fundamental part of the universe.
What makes me a bit scared is the big unknown.
Without any perception or relation to the physical world, and without memory or language, you wouldn't really "exist".