I am pretty sure you can find big pay differences when comparing only men with the similar education and experience.

You are proposing the model:

    wage ~ education + experience + error

If the errors are correlated with education or experience, that indicates a confounding variable problem. Most data shows that including race and gender as explanatory factors improves the model:

    wage ~ education + experience + gender + race + error

Until the coefficients of gender and race become approximately zero, one can say that the market is discriminatory. The counter-argument is that race and gender are not actually the real explanatory variables, but only proxies for better variables, like "ethic" or "determination". Those kinds of arguments are often viewed as racist and sexist.

According to my understanding, rimliu is proposing that

  wage ~ education + experience + gender + race + additional_factors + error

where additional_factors consists of other unknown factors (e.g. charisma, intelligence, appearance?). In which case, if education, experience, gender and race are kept constant, there is still going to be a fair bit of variance in wage caused by these unknown factors.

Either way, it would be pretty hard to model when these factors are far from orthogonal. Imagine the impact race would have on work experience if race is a big factor when hiring people.

Looking at it a different way, if productivity is defined as the combination of all relevant features (education, experience, ethics, etc.) then racism/sexism imply non-zero coefficients in

    wage ~ productivity + race + gender

This also assumes a competitive market where wage will asymptotically approach productivity.