- It's imbalanced classification: there are going to be more rejections than acceptances; so you have less statistical power, to find if there is a real relationship, when you just analyse the acceptances. Even apart from that, class imbalance can unintuitively affect post-hoc inferences.
- Scores are in context of the other candidates at a given level, and stage in the interview process, so are fundamentally already residual in some sense.
- I'm willing to bet, from hiring I've seen, that scores are distributed somewhat about the 'hire/no-hire' bar, and that extremes are rare - the process is going to be tuned, implicitly in the organisation, to make that 'hire/no-hire' decision, rather than predict performance.
> it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well
I think that's a very strong statement: it's not a random sample, its explicitly concentrated at one end of the distribution.
I do accept it is evidence, just not near enough to justify the conclusion the whole process is arbitrary.
It does not automatically mean this, but it is a strong indicator. If the output of the classifier is not correlated with a given output sample it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well.