Are you comfortable applying this same bayesian logic to other factors like race?
Secondly, and more importantly I think, what effects does a 'correct' bias have when it starts interacting with other people. It might be the optimal choice in isolation. But that doesn't make it a good idea game-theoretically if everyone starts doing it.
The end result can be difficult to explain though. Because it means that not taking certain useful information into account leads to better decisions.
I think that, sometimes, we argue that a systemic bias is stronger than the statistics warrant. When sometimes the argument should be that sysyemic bias, statistically warranted or not is harmful.
(Note, my original comment was me making the mistake I argue against in this post)
How someone looks informs your prior probability of who they are. What they say informs your posterior probability, but you gotta take that prior into account when forming the posterior. At least, if you want 'accurate' statistics.
(Bias - variance trade-off not withstanding, nor second-order effects of doing the 'correct' thing being possibly more detrimental than the advantage given by the 'correct' thing)