How someone looks informs your prior probability of who they are. What they say informs your posterior probability, but you gotta take that prior into account when forming the posterior. At least, if you want 'accurate' statistics.
(Bias - variance trade-off not withstanding, nor second-order effects of doing the 'correct' thing being possibly more detrimental than the advantage given by the 'correct' thing)
Are you comfortable applying this same bayesian logic to other factors like race?
Secondly, and more importantly I think, what effects does a 'correct' bias have when it starts interacting with other people. It might be the optimal choice in isolation. But that doesn't make it a good idea game-theoretically if everyone starts doing it.
The end result can be difficult to explain though. Because it means that not taking certain useful information into account leads to better decisions.
I think that, sometimes, we argue that a systemic bias is stronger than the statistics warrant. When sometimes the argument should be that sysyemic bias, statistically warranted or not is harmful.
(Note, my original comment was me making the mistake I argue against in this post)
Should is they keyword and maybe one day we'll reach that point. Meanwhile, looks and appearance can make a huge difference, so why take the chance?
A hobo who hasn't showered in a decade should be able to argue his case, be heard and receive justice. Anything less, you're using rules to enforce a social order, the legal system has no relationship with justice if this is acceptable.