I used to really like the idea of take homes, then I had kids and discovered just how hard it is to find time to do these (particularly for multiple applications while working).
Don't get me wrong, I think that they're generally a better test for the actual role, but they're not perfect, and they're game-able by spending way more time on them.
Interviewing is hard. Like, ideally, we'd just centralise the administration of these tests and you'd do a few every year (so formal CPD rather than an expectation that people do it in their off time) but that is never gonna happen in the tech industry for various good and bad reasons.
I would take an hour of live coding any time over a take-home task that is one hour only on paper and takes you three in reality.
Engineering is collaboration, and it's fair to expect candidates to talk, solve problems, and explain their solutions.
Where we could do better as an industry is the type of problems we give to people. I'm not a fan of LeetCode-style questions, especially when multiple ones are asked. Something closer to Earth would be better. But even if you have to ask something algorithmical, I'd prefer the style of "Advent of Code" exercises where the same problem has variations and multiple levels of complexity.
That's how Amazon does their interviews.
Another one I like was having to code review a repo which had specific issues in it.
Then add some new features.
Then in the final stage of the interview you go through it.
Nope.
100,000 applications. In under a single day.
With that kind of applicant pool, I’m honestly not sure what the best approach is—even though, in a perfect world, your suggestion would be the more appropriate route. The reality, however, is that these numbers are just absurd.
You should see the number of people that apply to college.
Start interviewing people, and once you have enough people who pass your bar, you tell the rest tough luck.
Secretary problem.
That said, your comment reminds me of a Monte Carlo algorithm I think I’ve heard about. There is a way to have some statistical confidence in getting the top K out of a sample of N without examining all N samples. I’m blanking on what it’s called, and I think it’s related to Reservoir Sampling. I don’t know if I read this or am making it up, but my instinct is that you can get to high levels of confidence after looking at sqrt(N) samples.
Say you want to take 10% at each round. You have to do this three times to get 100 people, and then another round to get the top 30.
If you tighten the criterion, noise matters more, so you will drop your actual best candidates, since people need a high skill+noise. But if you try to keep the unlucky "good" candidates by having a wider band, you pay more.
Also, if you are successful at finding the best candidates in a batch, noise matters even more the next round.
I've been thinking about how to vizualize this. I have it in my mind and I'll try to describe it.
You plot a bunch of points with S, N for each candidate. They are independent, so a scatter plot looks like a 2D gaussian. Lets say skill is vertical, and noise is horizontal.
You want to find the 30 highest points, but you aren't allowed to just look on the scatter plot and select the top points. You have to draw a line S + N = c, and choose the person with the highest c. Basically sliding a ruler out that crosses the S and N axes as far away from the origin as possible.
Observation: if N is high (wider distribution) compared to S, you just get the luckiest people. If S is high compared to N, you just get the most skillful.
If you want to reduce noise, the way to do it is to have more independent measurements (interviewers), not to stop interviewing early.
The good news is that there’s no such actual thing as “best” in this situation and people have many dimensions, they can’t be ranked perfectly, but they can still be ranked approximately. We also don’t need to get the exact 30 people out of 100k people with infinite precision, we will get an amazing set if we can take a random sample of the top 1000 candidates out of 100k candidates. Having 100k candidates gives us the opportunity to end up with a selection from the top 1%, say, whereas if the number of applications was 40 people for 30 jobs, you might be stuck accepting people who are below average.
Instead of a "noisy" sieve, you're just using the time of application as your sieve. Which, unless your job opening values the skill of submitting lightning fast job applications, is pretty much 100% random noise.
You say a sieve approach drops the best candidates. Your approach also drops the best candidates.
And if the bar is actually so high that only the best candidate(s) qualify for the job, then your approach would imply you interview an expected half of the applicants before you fill the positions.
It's pretty obvious a "noisy" sieve (with some signal) is better than sieving by application time or sorting by arbitrary order and taking the top N applications. You don't have a magical solution. The people handling those 100000 applications are not that stupid.
If you're already reasonably confident from the first batch of people what the potential in the pool is, why would you wait? People have work to do and they want those people in as early as possible.
I'm not saying you will do much better sieving by time, in terms of quality. But you will save all the effort of looking at the people at the back of the queue, for little loss.
Sure, half the best people will be at the back, but what is your loss on taking the second best people earlier? Chances are you won't even be able to tell.
In this particular case, MS didn’t have to wait for people to show up, they got 100k candidates before they could have even made a decision on a small subset. And in general, the problem being discussed in this thread isn’t having to wait, it’s how to deal the flood of applications coming in too fast.
The sieve approach does solve the same problems you’re talking about, while not adding noise or being arbitrary or capricious with candidates by having to reject them randomly. The sieve saves effort and time spent evaluating candidates. Don’t forget the idea is that the top of the funnel is criteria that are both low bar and much faster and easier than an interview. In the case of 100k applicants, the top of the funnel should probably be automated and measure things that are as easy to measure and objective as possible like school grades, amount of work experience, maybe test scores, etc. To whatever degree possible, with 100k candidates, you can probably automate whether the application meets some minimum criteria and easily rule out most applications.
Only the bottom of the funnel takes time, and you only use the bottom of the funnel on maybe 2x or 3x the number of job seats you have available. This way you only spend the same amount of time evaluating candidates that you would have use any sampling method.
Regarding noise, the sieve is a better approach than early stopping because the criteria at each layer are not subject to or sensitive to the same amount of noise. Because the bar is a minimum criteria at the top of the funnel, noise in that selection process is far less likely to reject top candidates.
BTW, the Secretary Problem’s 37% rule is only applicable to picking 1 single candidate, so you need some kind of modified algorithm to pick the best 30. Also that solution implies you have to fully evaluate each candidate before evaluating the next candidate. This is potentially much costlier than using the sieve approach, because you have to interview all the candidates that you reject. The benefit of the sieve is based on the observation that we can use extremely easy to measure minimum criteria to partial evaluate candidates for the purpose of culling.
Your proposed alternative is absolutely subject to the same problem, AI-written code and LLM-written mental process. Think more carefully about whether it solves anything and exactly what problem it solves, because it does not solve the problem of people trying to game interviews with AI, your suggestion is potentially worse on that axis. The whole point of doing it live is to get to the talking part that you suggested more quickly.
Wouldn’t you rather be discussing your thinking immediately and get a sense for what the real questions are and how you’re doing, and have the opportunity to talk about it, than waste several hours doing it at home trying to write something up without getting clues along the way or being able to ask questions, only to have the next guy get hired because they used AI, or because they’re fine with live coding, or because they’re better at writing than coding, or because they got lucky?
You have to live code on the job. Do take a moment to consider what the hiring manager needs and whether take-home problems are either more effective or more efficient for the company. Managers want to find the best candidates, and best involves more than coding, it involves communication and collaboration skills too. Live coding can give some indicators of those things, and take-home problems cannot.
I believe live coding is a scourge and we should get rid of it altogether.
Get a tiny open ended project, which requires more design/architecture skills than actually finishing the solution. You do it at your own time and shouldn't take more than 1-2h. You submit it and provide your mental process in writing.
We then hop on a call and we can dicuss it further, talking about possible issues and what to do to get it released if a requirement changed last minute.
If you can do that, come across amicable, knowledgeable and experienced enough given the seniority of the role, and feel like you are not reading from an hidden ai window then you pass.
"Exciting casts" are absolutely worthless in real life, everyone has to balance shipping vs code artistry.