I wasn't taking sides on whether it's physical or not, so I wouldn't particularly agree with your statement.

I'm only trying to say that if you don't accept that consciousness is physical, you aren't accepting the symmetry between you and other minds, and can't reason from the assumption they have the same experience.

Once you contradict the known fact that you experience your own thoughts and not others, you descend into incoherence. It seems to me kind of like the issues people have with quantum weirdness. People are accustomed to assuming they know things for the sake of argument, but assuming you know the unknowable can sabotage your reasoning.