I guess my point is we're talking about unconscious bias, and your story seems like it could be an example of just that. If it's unconscious you wouldn't know -- we can assume something is racist that really isn't.
If we can't know, how can we approach these situations and be sure we're without bias; particularly if we want to call others out for their [apparent] unwarranted biases.
An example, a neighbour is a bit discomfiting to me, my wife refuses to be alone in his presence, but he's Polish; so is she xenophobic, or is he a jerk (or both I guess). He's the only Pole in our street, an outside observer could easily conclude she was being xenophobic.
Well-dressed is "over-dressed" if everyone else is wearing t-shirts.
If everyone is in t-shirts and skirt/slacks and I'm "well dressed" in a suit and tie, I'd say I'm overdressed in the context of that situation.
To be fair though you brought focus on how he was dressed when it sounds like you should have said "he was dressed normally for our office". It really came across as if he stood out because of his choice of clothes.
Which is of course the pattern of these discussions. In exactly the same way my intern got cross-examined for being black in a white space, any mention of racial bias gets cross-examined because white people are very eager to defend the honor of white people. How was he dressed? Was it too rough? Well then it must be that he dressed too well! Never mind that all of the other people dressing well don't get stopped. And this cross-examination happens despite the fact that the sort of racial bias described here is in no way rare and is demonstrated in all sorts of formal research and oceans of personal anecdote.
It was his judgment, my judgement, and my boss's judgment, that this was an unconscious racial bias incident. Neither HR nor the black employee group thought otherwise, and they deal with this stuff all the time. Is it possible we're all wrong? Sure. Is it possible that the anonymous white person was actually a space alien beamed down and probing our defenses? Sure. But based on the preponderance of the evidence, we decided what we decided.
If that's not good enough for an anonymous rando, that's fine by me. If you'd like to learn more, read any one of the books I've mentioned in this thread. Preferably DiAngelo's, as it goes into detail examining exactly the type of reaction you and others display here.
Absent fMRI recordings from the elevator lobby, I of course cannot prove that it was unconscious racial bias. But we can't prove most things in life with that level of evidence. Nevertheless, we have to operate in a complex and dynamic world. This is my best interpretation of events. It was also the best interpretation of the person present, of my boss, and of several other people in the company I discussed this with.