This is wrong. Apparently google has internally done much research and their interview techniques have strong collreation to success on the job.
With that said, the large variation in ratios of disadvantaged minority employment at big tech companies does seem meaningful (and he cites that wide variation). I think intern and HR pipeline variation is likely very relevant to this difference.
The problem is with his implication that throwing darts to pick from a pool of moderately qualified people would yield a reasonable result.
Meanwhile it's crucial and nigh-impossible to identify how much success is based on interview performance as opposed to selection effects in who interviews, and how interviewers see different candidates.
It does not automatically mean this, but it is a strong indicator. If the output of the classifier is not correlated with a given output sample it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well.
- It's imbalanced classification: there are going to be more rejections than acceptances; so you have less statistical power, to find if there is a real relationship, when you just analyse the acceptances. Even apart from that, class imbalance can unintuitively affect post-hoc inferences.
- Scores are in context of the other candidates at a given level, and stage in the interview process, so are fundamentally already residual in some sense.
- I'm willing to bet, from hiring I've seen, that scores are distributed somewhat about the 'hire/no-hire' bar, and that extremes are rare - the process is going to be tuned, implicitly in the organisation, to make that 'hire/no-hire' decision, rather than predict performance.
> it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well
I think that's a very strong statement: it's not a random sample, its explicitly concentrated at one end of the distribution.
I do accept it is evidence, just not near enough to justify the conclusion the whole process is arbitrary.
Enough for what? Why not? They specifically have a "score", not just a hire / no hire decision. Sure it would be better, but you don't make any convincing argument why their analysis is not valuable.
> designed to be a Classifier, typically tuned with an emphasis on having a low false positive rate.
classifier of what? Why is classifier capitalized? I don't think so.
Let me throw out some made up numbers as an example.
Company X hires 100 people out of 500 applicants.
If only 30% of them perform well, you might argue the interview process is rubbish.
However, if you can test and show that of the "un-hired" 400 only 5% would perform well, then clearly the interview process is better than picking at random.
This is why just looking at hired performance is not indicative of whether the process works - because the task was not just to predict good performers but to predict a better rate of good performers than a random sample of applicants.
>One of the open secrets of working in technology is that technical interviews are completely worthless as a predictor for whether someone is a good hire or not. [...]
>> We looked at tens of thousands of interviews, and everyone who had done the interviews and what what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship.
> It’s amazing to think that Google found zero relationship between an interviewer saying “Hire” and whether the candidate was actually a good hire or not.
The fact [the scores of someone who gets hired] are not predictive of their eventual performance, does not mean that hiring process is completely random, or worthless.
In order to tell that, you would need to look at the cohort of people who were 'no hire' and compare them to the 'hire' cohort; it's not enough to just look within the 'hire' cohort.
In other words, the hiring process is (1) designed to be a Classifier, typically tuned with an emphasis on having a low false positive rate. Hoping that (2) the scores of the small subset of candidates who are classed 'hire', also act well as a regression to the candidate's eventual performance, is a big additional ask.
Just because there is evidence that (2) is untrue, or particularly noisy, does not necessarily mean that (1) is broken.
I don't see a great way of solving this, without a large company like Google also randomly hiring 'no hires' from different stages in their hiring process (and subsequently measuring their performance) in order to quantify the performance of subsequent steps in the hiring process - not aware of anyone who has done that; it would be really interesting to read how a company who does that gets on, but an expensive experiment.