Forcing people to work with someone against their will or preference also causes harm, and if society wants to do that it has to justify that the harm prevented is significantly greater than that - which is true for some types of discrimination and not true for others.

If nobody in the village wants to work with Bob because he's an asshole, they should be able to discriminate against Bob (and other assholes) as the social harm of Bob being unemployed and unemployable does not justify the social harm caused by the state coercing people to work with Bob, violating their freedom to choose. It would be harmful (and IMHO immoral) to prohibit discrimination against assholes. Freedom of association - and the freedom of not-associating it implies - is not inherently hateful, it is one of fundamental human rights that should be curtailed only if other alternatives aren't feasible - merely having some reason or some harm prevented is not necessarily enough.

Well, "I don't know if I believe anyone is 100% a dick-". Bob may be an asshole precisely because he's being discriminated against and that makes him angry. Our society seems to think that even the worst people, even murderers and thieves and rapists, can be rehabilitated and included.

But I see your point.

And I'm not arguing that no one should ever be allowed to discriminate for any reason. I'm simply flipping the script: instead of the default position being that discrimination is acceptable, discrimination should require justification.

Someone might be willing to publicly say "we don't hire assholes" and defend it. Go for it. It would be much harder to say "we don't hire short people" or "we don't hire dalits" and defend that.