Of course we can be all conscious but with different qualia. But the thing is that if consciousness was merely a physical phenomenon, we should be able to determine qualia purely by looking at the physical system, just as we determine any other property. However, we could be in a situation where we have understood our behaviors down to particle level, and I would still have no idea how you perceive colors.