I don't like very much this reverent way of thinking about consciousness, as if it is from another world, or a different essence.
I believe consciousness is the ability of the agent to adapt to the environment in order to protect itself and maximise rewards. It's not just in the matrix multiplication, but in the embodiment, the environment-agent loop. Consciousness is not something that transcends the world and matrices, it's just a power to adapt and survive.
And it feels like something because that feeling has a survival utility, so the agent has a whole neural network to model future possible rewards and actions, which impacts behaviour and outcomes.
Why not? Those things have zero-consciousness that is conscious only of itself and which correctly reflects their lack of self-model.
Obviously other living humans have the highest similarity, so they are automatically deemed conscious. Next are other primates, followed by other domesticated mammals, and other animals.
Furthest from the status of conscious are creatures we see as automata like dung beetles rolling their food, or jellyfish ... jellyfishing.
Presumably we'd apply a similar process to hypothetical AGIs.
It is more likely that consciousness is just a property of brains inside bodies. The interesting question to me is the level of complexity of brain and body required to produce something like what we experience and what is it like in other arrangements of brains and bodies.
I also don't know that intelligence is the great thing that we think it is. It's an adaptation. It's an adaptation that lots of other organisms survive just fine without.
We are all p-zombies. Problem solved.
The longer we refuse to acknowledge that consciousness is nothing special, the longer it will take to tackle this topic. The only reason we cling to the idea that our minds are somehow special compared to other animals of various complexity is because we refuse to acknowledge that consciousness might exist in something we can't communicate with and that consciousness is a sliding scale rather than a binary property. Ascribing consciousness only to ourselves is hubris.
If one spends a bit of time observing humans, they will inevitably realise that some humans are more 'conscious' than others also.
tl;dr: we're all p-zombies. The fact that we think that each one of us isn't individually doesn't detract from that.
Just as natural sciences left less and less hiding places for god to exist, ML is leaving less and less hiding places for this borderline magical version of human-unique consciousness to exist. Answering this question in any more detail requires a much more rigorous definition of consciousness which is a big can of worms in itself.
If I made a list of everything in order of how certain I am that the item on the list exists, consciousness would be at the top by far. Everything else could just be a nice illusion.
To put it another way, if AGI is computable, then we are all p-zombies. And evidence is starting to strongly hint that AGI is computable.
Why do you think the burden of proof should be inverted? The mere fact that most humans intuitively feel "something" doesn't count for much of anything, especially once you stipulate that p-zombies would vote the same way.
So, you can argue something like our brains are nothing but curve fitting machine with enough parameters, but then you are probably forced to argue that consciousness is very closely related to computation, to the point were a coin flip, or a hello world program has some sliver of consciousness.
There are of course two possible ways around that, either one can argue for p-zombies, that is intelligent but not conscious beings, which then seems to require a super natural explanation for consciousness. Or you can argue that the brain is different, and that this gives rise to consciousness, and to general intelligence, which is the explanation that at least corresponds most closely to my subjective experience (but that is precisely what a soulless machine would write, isn't it?)