Which becomes even worse when realizing "false positive/true positive" is too binary of a definition for a spectrum such as job performance.
And I say this as someone who's in favor of giving people a quick check on whether they can actually do FizzBuzz.
The only way to provide this evidence would be to to conduct an RCT in which you hire some percentage of people who fail the test as a control group. The nascent psychologist in me loves this idea, but it seems unethical, and presumably this is why nobody does it.
But I disagree that we need ”evidence.” It’s flat out obvious. If you give your job candidates a discrete math problem orders more difficult than the usual ticket, you minimize the risk of hiring someone who can’t do the job.
You're also maximizing the odds they will get bored and feel baited. With all due respect, you could have picked a better example to illustrate your point.
This is exactly why you can't make such bold claims. You're looking at this from one perspective without considering all the variables which go into the workplace equation. So far, the best correlator to job performance is giving people work corresponding to what they will be doing at the work place. For various reasons, this isn't really happening in software development, making people all to eager to grasp at proxies which they swear will do the same thing. Maybe whiteboarding is in line with the day-to-day at FAANGs, but tons of companies ask questions which are not in line with the day-to-day at all. Both vertically and horizontally. It creates dissatisfaction, burnout, bad cultures, sometimes even animosity, as if hiring managers are playing pranks on candidates.
If you're only looking at this from the perspective of "get someone who can do the job", you're going to miss the forest for the trees. As others say, the company is also being interviewed.
I like this point. This is an especially big problem in my area (data science), in which people think they are being hired to do deep representation learning, and are subsequently mortified to learn that the job is principally about transpiling ”product manager” to ”SQL”. The cure is to be up front with job candidates. Tell them the job involves no deep learning. Explain that you give hard interview problems because you need to know that they can reason about eg asymptotic complexity on the rare times where it arises.
> So far, the best correlator to job performance is giving people work corresponding to what they will be doing at the work place.
I don’t necessarily disagree, but I believe you want to optimize for the hardest day rather than the average day. If your team is optimized for who can best solve an average problem, you will eventually have a problem that nobody can solve, and the business will suffer greatly as a result. Hence the discrete math problems.
I think this is a huge mistake. I work in data science, and nearly universally my colleagues are motivated by opportunities to learn. If you're testing for the very rare day, so that your employee has the right answer in their front pocket when that happens they are never going to feel challenged, feel like they're growing, and they're going to jump ship.
I think you're doing snobby filtering, and it's not to your benefit.
On the hard days, the answer has never been seen before, and therefore can’t be in anyone’s pocket. You need to know that you have employees who are smart enough to discover it. Technical interviews are how you find them.
The observation that your employees are motivated by learning opportunities is useful for how you manage them, but this doesn’t tell us how to hire people who can do the job.
There are a few problems with this. If your "hardest day" deviates far from your "average day", while being very infrequent, you're still not doing a good job enticing the candidate. Worse, by looking for people who have already acquired the skills to combat the worst you can throw at them, you're severely stagnating their growth.
Then comes the cherry on top, if your company is doing it while not providing absolutely stellar benefits to function as external motivation, you're further incentivizing people to work on ditching your place. Many companies do not give total comps in line with the skill level tested for, setting themselves up for failure in this regard. Altogether, you're still looking at this primarily from the job interviewer's perspective, not the perspective of the candidate which translates into the company's long term (lack of) health.
>you will eventually have a problem that nobody can solve
No. Someone has to be the first person to solve a question. This strategy doesn't take into account the capacity of individuals to learn and overcome challenges. What's more, most individuals will not be on the highest level, allowing mentors to train them to higher levels.
>I don’t necessarily disagree
It's not about disagreement. Schmidt and Hunter did a meta-analysis on this way back in 1998, and psychology has yet to find the silver bullet. Look at table 1[1]. Most studies checking the market find similar things. It is ridiculously difficult to find correlations between job performance and selection criteria. All corporates, including FAANG, are not exempt from making the very human mistake of doing something which seems great, but ends up being part of survivorship bias. In something as multi-faceted as the job market, combined with the young and everchanging nature of software development, making statements on correlations shouldn't be taken for granted.
Notice how the study above points out years of experience as one of the worst predictors. Most of the job market still uses YoE as if it is gospel. The market is nowhere near as rational as people like to believe it is.
[1]: https://www.researchgate.net/publication/232564809_The_Valid...
You might enjoy Work Rules by Laszlo Bock. They actually did do this at Google to validate their interview approach.
In all seriousness, thank you for bringing this up - it’s an interesting story and I will probably look into it. But I also think it’s pretty obvious that coding interviews work extremely well, and any discussion of ”how can we create a procedure that is better for everyone” that doesn’t acknowledge this is hopeless.
Have you ever seen a hiring process that's simply not effective?
The undercurrent of this irks me a bit. It assumes a kind of omniscience on part of the employer, and enough time for them to develop a good interview process. I think this is really hard to do.
I think you're generally right. And a lot of the time, employers are empathetic, pragmatic, self-aware and really are doing us a favor when they deny us. Some take into account I-O psychology stuff, too.
I (and others probably) are eager to hear what companies you feel do a good job at hiring.
> Everyone realises that programming interviews produce lots of false negatives.
Let's assume a bad scenario. A company can disregard evidence of competency, like having a portfolio, to basically have a rolling, year round tech Olympiad. To them, it's about fronting as a strong candidate in events, even at the expense of being good at the role.
You can be a pro at pole vault or discus throwing, but a poor carpenter. But there's a mentality out there - with some - that a strong, fit candidate can handle any technical challenge.
Can a chemist be a great chef? I bet, I also bet some could overthink and/or overengineer preparing a simple meal. Does the interview process do anything to check for soft skills, like curiosity, simplicity, etc? Hopefully.
> The goal is minimizing false positives. Failing to hire a strong candidate carries a much lower cost than hiring someone who can’t do the job.
I think some companies - not all - can create a "squid game" contest out of candidates. Worse, I feel candidates can unduly suffer and be humiliated in these cases.
The company gets a superb interviewer, who is no doubt a very intelligent person, but not necessarily one that can adapt to technical reality, contingencies, ambiguities, etc. Some interview processes end up focusing on theoretical and rote knowledge, not integration and synthesis, when they deeply need that. Turnover happens.
But we're back to the beginning, did the company: 1. really know what they needed for that position, 2. interview process check for that? My guess is when you wrote that you were assuming situations where both were true?
Hire for strengths, not to avoid weaknesses.
This is true. They are closer to ”the rare, most intellectually difficult parts of the job.” A hockey player typically spends 70% of each game sitting on the bench; yet this isn’t what is emphasized in tryouts.
> Hire for strengths, not to avoid weaknesses.
I guess you didn’t read the part of my comment where I wrote:
1. ”Failing to hire a strong candidate carries a much lower cost than hiring someone who can’t do the job.”
This is the default position of large tech companies who do coding interviews, and especially those like Google and Microsoft who interview with very difficult discrete math problems.
If we assume 1, the hiring practices of Google/Microsoft make perfect sense. They are not trying to hire all the extremely talented candidates. They are trying to ensure that all the hires are extremely talented. Put differently, they are optimising for precision rather than recall.
It seems like you disagree with 1. And if you’re right, then you have a huge opportunity, because this suggests you can disrupt any vertical where the incumbent does the LeetCode thing.
But my default position is scepticism, because if it was possible it would probably already be happening.
Only if you view hiring as the competitive advantage of Google/Microsoft/Apple. But everyone cargo cults the hiring practices of Google/Microsoft/Apple without becoming Google/Microsoft/Apple, which shows that it's not actually some magical unique competitive advantage.
We can get into a discussion/argument about what is the competitive advantage of Google/Microsoft/Apple (which collectively own near 100% of the market for mobile OS, desktop OS, web browser, search, etc.), but that's kind of a tangent IMO.
I will say that Google/Microsoft/Apple have a competitive advantage specifically in hiring (that is, getting many of the top candidates to apply) because of their prestige (a product of the aforementioned market dominance) and vast wealth. The latter in particular allows them to offer more compensation, especially stock compensation, than other companies.
It's important to mention though that the formalized hiring practices of these companies are largely ex post facto and could only be put in place after they were already successful. It's unlikely they hired exactly in the same way during the early years. Steve Jobs was introduced to Steve Wozniack by a friend in high school! Nobody can plausibly claim that the success of the Apple II was due to "coding interviews". Larry Page and Sergey Brin met in college, where they were already working on the ideas that led to Google. Hiring doesn't just turn into success, that's not a reliable formula.
You can set the bar as high as you like, but you won't be able to hire anyone if candidates just walk away from your obstacle course. Talented people put themselves through a lot of crap to work at BigCos because there's a big payoff working for a BigCo. That's what tends to make the BigCo hiring pool better than for other companies, not the hiring bar itself. You can afford to be extremely picky if you have a wealth of options to pick from.
You're conflating some things here. Without unpicking it all, the main thing I'd say is that you've only demonstrated that amazing hiring is not sufficient for competitive advantage. You have not demonstrated that it's not necessary.
I'd say there's a big difference between a company finding engineers and engineers throwing themselves at a company. The latter is what often occurs with BigCos. They have a big pool of candidates throwing themselves at the company, and the company gets to pick whomever they like from the pool. How much credit to due to the company for picking? Whereas I think you'd give a company more credit for going out and specifically recruiting/poaching one person they want, even if that person wasn't applying for a job with them.
Lots. Firstly for becoming a place people want to work. Secondly for sifting through all of those people.
Ideally without an increase of time/cost requirements.
The reason people are upset by this issue is that it highlights a misalignment in the incentives of employers and job seekers. And with this I fully empathize: job hunting is hell in any industry, ours included.
But we can’t begin to have a conversation about improving the situation until we have a rudimentary understanding of what is happening today. And all these ”Google doesn’t know how to hire smart people” takes are brutally handicapping the discourse.
Well, I'd like to have a word with the Apple Mac Mail app team. And the Disk Utility team. And some others...
I look forward to seeing you disrupt every major tech company in the world via your superior hiring strategy.
> EVEN IF THE CANDIDATE CAN IN FACT PROGRAM JUST FINE.
You don’t seem to have thought very hard about what employers are optimising for. Everyone realises that programming interviews produce lots of false negatives. Even the strongest programmers have bad days. From the employer’s point of view, that’s okay - minimizing false negatives is not the goal. The goal is minimizing false positives. Failing to hire a strong candidate carries a much lower cost than hiring someone who can’t do the job.