I find that a sometimes simple questions like this can turn into people calling me a SJW (then dismissing everything I ever say), "your the racist for accusing me of being racist", people thinking that I'm being mean to them, etc.
Also, what exactly are you saying? I can't see how "no, why do you ask" said with a smiling face in a friendly tone could be construed as either aggressive or SJW, but then maybe I don't live somewhere where people are incredibly touchy.
The evidence shows that almost all of us, including members of minorities, are conditioned to have racial prejudice.
The reason this is a problem is that it systematically puts some people at a disadvantage based on their appearance, even though the evidence also shows that within group differences are greater than between group differences.
Unconscious bias can be overcome by making it conscious.
We are all racists. We can overcome the problem if we face it.
Trying to protect people from having to consider whether their actions arise from unconscious bias is actively working to perpetuate the problem.