Hell, I have worn a suit to a friend's place and gotten hate from it.
I've had several internet fashionistas try to tell me that fedoras (and bow ties, and suspenders, and etc.) are Ruined Forever because a few unpleasant dudes on OK Cupid wear them. I find the notion absurd.
You're reading their casual implication in the wrong direction. What they usually mean is that disgusting, reprehensible goblin implies poor fashion decisions. Thus, by Bayesian inference, poor fashion decisions are evidence which increases the likelihood that you are a disgusting, reprehensible goblin.
And fashion is entirely about Bayesian inference--or "judging a book by its cover," as it is sometimes known. ;)
(I will now pause to enjoy your apoplectic fit. Yes, you. You know who you are.)
By comparison, my last-minute no-budget "Devil out on the town" Halloween costume was a black suit, black shirt, black fedora with the King of Hearts in the band, blood-red tie.
Dear Internet Fashionistas: You've decided that fedoras and suspenders are the mark of a horrible person. Your opinions have no connection to reality. Over here in the real world, I look good. When I come back to the bar the next day to pick up my credit card, and the bouncer remembers me well enough to compliment my outfit from the previous night, that means I'm doing it right.