Of course. I've never seen someone argue they were being discriminated against on the basis of race, and that they know this because someone else offered to do more work for less money and were chosen.

I agree it is currently popular to assume "-ism" simply from disparate outcomes, but I don't think it's right (in both the "correctness" and "morally proper" definitions of the word).