That reason is the same for any marginalized group: "preventing the social harm done by X discrimination".
Why not simplify the statement and say "preventing the social harm done by discrimination"?
Accepting discrimination against some people is inherently a hateful position.
If nobody in the village wants to work with Bob because he's an asshole, they should be able to discriminate against Bob (and other assholes) as the social harm of Bob being unemployed and unemployable does not justify the social harm caused by the state coercing people to work with Bob, violating their freedom to choose. It would be harmful (and IMHO immoral) to prohibit discrimination against assholes. Freedom of association - and the freedom of not-associating it implies - is not inherently hateful, it is one of fundamental human rights that should be curtailed only if other alternatives aren't feasible - merely having some reason or some harm prevented is not necessarily enough.
But I see your point.
And I'm not arguing that no one should ever be allowed to discriminate for any reason. I'm simply flipping the script: instead of the default position being that discrimination is acceptable, discrimination should require justification.
Someone might be willing to publicly say "we don't hire assholes" and defend it. Go for it. It would be much harder to say "we don't hire short people" or "we don't hire dalits" and defend that.
It definitely is okay to discriminate over personal attributes, character and previous experiences, and employ the person who you want to work for you - which is not necessarily the same as the person 'most deserving of a job'. Freedom of association is an important right, people should be free to choose whom to deal with, and imposing restrictions on it (like the protected groups) is morally justified as an exception only if required by important reasons, for example, preventing the social harm done by racial discrimination in employment; in the default condition people should be free to refuse to associate with anyone simply because they don't want to.