“Best person for the job” is either a meaningless phrase or a myth, depending on how you choose to define best.

In reality, there are many comparably adequate people for getting any job done, and no clairvoyance that lets us know who among them will deliver the best outcome.

So we resort to picking among the comparably adequate candidates by either broadening “best” to include non-task factors like culture harmony, diversity of perspective, gut feeling, etc

That a bunch of adequate people all came from the same class at the same university is actually pretty plausible, and (subject to tradeoffs) those people will probably even work pretty well as a group because of that shared experience.

I’m not saying its the ideal strategy to only hire that way, but just that it’s not nearly so straightforward as you suggest.

At one previous company we advertised a job and got five CVs total from it. Anyone who's done hiring knows that the quality of five randomly chosen applicants is not going to be high.

I'm 100% sure that the best person for the job wasn't the referral we ended up hiring, but the real best person was too busy sitting cross-legged on a mountaintop meditating on the meaning of pointers and writing all their code in haiku to send us an application.

You don't get to chose from all the people, you get to choose from the ones your hiring funnel brings in. Sometimes the market is tight and you can't find anyone good.

It comes from risk aversion. Firing people is painful and expensive. And it's hard to know how good someone actually is just from interviewing.

So if I've worked with someone previously and know they're not lazy or incompetent, that's worth a ton -- even if they're not the absolute best person for the job.

In most cases, it’s sufficient to avoid the “people who are a bad fit for the job” rather than needing the “best people for the job”. Deep knowledge acquired via years of working/schooling with them is a much better selector than a half-day of interviewing.

The odds that a company truly needs the best ten instead of "anyone in the top 10%" are equally slim-to-none. A cohort of fraternity brothers may give up the theoretical maximum quality in favor of known competency (& weaknesses) here and now.

The odds that you can better assess "best person for the job" from a resume and a few hours of interviews vs. several years of working together are also slim to none.

OTOH not every job needs the best person for the job. Good enough will often be fine.

Also, I know people in minority groups who are fantastic at their jobs and have had year-plus job searches.

The "labor shortage" is bullshit. So many good people can't find decent jobs even today, and wages are still hilarious low compared to the outright robbery that capital gets away with.

That being said, I don't think it's necessarily because of conscious racism, sexism, or ageism that companies tend to end up redlined. It has a lot more to do with the fact that managerial decisions are made based on motives that have little to do with business at all. Silicon Valley ageism, for example, doesn't exist because middle-aged men think young people are all geniuses. (Trust me, we don't.) Rather, it exists because the subordinate's job is to make the boss feel young again, to be the Jesse to their Walter White. It has nothing to do with the needs of the business, but it also doesn't get in the way of the business, so it's not likely to be changed.