I agree with your general premise, in that there are bad actors, and appearance is a powerful classifier, so identifying potential bad actors by appearance is genuinely useful. I think there are many caveats in practice, such as:

  How do I demonstrate that I arrived at a conclusion reasonably, with data?
  How do I calibrate my probabilities, instead of a binary "safe or unsafe"?
  How do I keep from overanalyzing appearance and making incorrect perceptions?
  I think the primary sign of danger in your example is being in a dark alley.

Moreover, learning danger where there is danger is valuable, but so is unlearning danger where there isn't danger. And then there are the errors of learning danger where there isn't danger, and unlearning danger where there is danger. So, I take your point broadly, but there are many demons this way.