If you want to make it more of a fair consideration of time, consider moving your take home to interviews, that way there isn't a time cost asymmetry. You can enforce your "20 min max" claim this way, you can judge a candidate's performance, thought process and filter out anyone who is LLMing or spending inordinate amounts of time on them.
You will also make a better impression on candidates by investing your time in them in the same way they are with you. Maybe you're hiring kids out of college without experience, but you only have to do so many take home tests before you realize that they're a waste of time, and pass on potential employers who throw them at you, or you learn to just send them your hourly rate for the test.
There is usually a huge disconnect between someone who knows that “this task should take 20mins” and doing it cold in a super high-pressure environment.
People sweat, panic, brain freeze, and are just plain out stressed.
I’ll only OK something like this if we give out a similar but not the same task before the interview so a person can train a bit beforehand.
I’ve heard it all justified as “we want to see how you perform under pressure” but to me that has always sounded super flimsy - like if this is representative of how work is done at this organisation, then do I want to work there in the first place? And if it isn’t, why the hell are you putting people through this ringer in the first place, just sounds inhumane.
If you give unlimited amount of time, you're giving an advantage to people with no life who can just focus on your assignment and polish it as if it were a full time job.
If you give a limited amount of time, then you're making the interview a pressure cooker with a countdown clock, giving a disadvantage to people who are just not great at working under minute-to-minute time pressure.
The ones we use have a clear scoring system and prepared inputs - all it matters is the generated output.
I started refusing take-home tests a couple of decades ago, but when I did them, this is 100% what I would have done.
In my experience this is the wrong game theory. Unemployed people can make job hunting their full time job, so a 20 minute take home doesn't select for "who delivers the highest quality solution in the least amount of time," it selects for "who is the richest applicant who can burn hours on a take home to deliver a higher quality result than people with less time they can afford to spend?"
Also, nobody should ever self-select themselves out of an interview process. Passing a resume review and getting a callback is about 10% likely: for every job hunt, in my experience , candidates get about 10 callbacks for every 100 resume sends. From there, it's about 20% chance to get to final stage, and from there, maybe 50% to get an offer (you're either their first choice or second; if second, your hiring hinges on whether the first choice accepts). Math is right there: once you pass a resume check, in terms of the volume of applications you've sent, it's optimal to spend far more effort into this gig than into firing off ten or twenty more resumes.
Therefore, even if the candidate doesn't think they're a good fit, they should do everything they can to stay in the game, including lying by omission.
After all they might be engaging in imposter syndrome, right? Why assume for the interviewer that your python skills aren't good enough - maybe the interviewer understands perfectly well that you've only used it for scripts and one off tools, but doesn't care because they personally believe your startup experience is more valuable to them and they believe you can up skill! Maybe the take home was designed poorly by someone who was tasked randomly by a lead to shit out a take home, and it's not an accurate indication of what the job would be like. Maybe they sent you the wrong take home? Maybe it's a good take home but you need money so fuck it, if you manage to sneak in despite not being a good fit, you can just bust ass to upskill and make up the difference before anyone notices. Or fuck it twice, it's a shit market and who knows how much longer you'll be able to sell your labor as an engineer, even if you can only fool them for two weeks, that's two weeks of income while you still keep up your job hunt.
GP specifically stated that this was the point of the takehome though. If the person handing it out specifically warns you that struggling with it means you aren't a good fit then if you struggle with it that's not imposter syndrome - you aren't a good fit! Not dropping out at that point is just refusing to acknowledge reality and insisting on wasting everyone's time.
Sure, but, there's nothing incentivizing the candidate to not waste time. At the very least, they can get free interview practice.
I get that that's very annoying when you're on the hiring side, but it's not all so bad in the end, you're getting paid for your time!
People who have really good jobs aren’t applying for jobs below them, so your applicant pool will always be people who are in an equal or worse position than your job.
No one at, Anthropic, for example, is applying to a job at Geico.
Also I know plenty of people in startup world who are phenomenal engineers that only have companies I've never heard of on their resumes - startups that for one reason or another simply didn't have a news-grabbing exit.
But they’re not. And they won’t. And that is my point. They’d make a ML Engineer post on LinkedIn and get a bunch of people for whom Geico would be a step up.
There will never be a job opening from Geico that someone at Anthropic would apply to.
That’s my point - your pool will always be people who are in a worse position than your job. Being laid off is a worse position than a job.
You’ll never see Anthropic candidates in a Geico hiring pool, unless they were laid off for being lousy and can’t find anything else.
The market is pretty efficient - people wouldn’t bid for jobs that are worse than their current situation.
This still seems like an oversimplification. It's easy to label FAANG, "frontier AI companies," whatever else, but the vast majority of jobs and the vast majority of engineers are in a soup that's maybe able to be split between "startup world" and "enterprise world" but beyond that, difficult to say one is "worse" or "better." And I've worked alongside FAANG people in startup world so, either that isn't a "worse" job and therefore your theory doesn't work because that means it's not really possible to accurately evaluate every single company as objectively worse/better, or, your theory doesn't work because people do apply to "worse" jobs.
Take home tests were never a worthwhile signal. Pre-AI, people would search for solutions or have another person complete it.
Employers can, with a somewhat high (?) degree of certainty, weed out candidates using AI.
The AI point is worth diving into a little. This was a year ago, so SOTA was worse, but I didn't find it terribly hard to write questions AI couldn't solve, whose answers you couldn't search for, and which good candidates could solve. The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section.
I don't think that moat will exist indefinitely, but today's AI just isn't very good at a lot of incredibly basic tasks unless the operator has enough outside knowledge to guide it in the right direction (and if a candidate did that I mostly wouldn't care because, by definition, they had the knowledge I was looking for). I use AI a lot, it's great at a lot of things, some even quite complicated, but it was weaknesses, and those are pretty easy to exploit.
> The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section
I also like how you allow/encourage self-assessment, where if a candidate can't do the test in ~20 minutes under zero pressure, they probably won't be a good fit in the role itself.
If you’re some random startup or no name company, I don’t bother. I already have a good job.
If you’re a top name hot company offering $600k+ total comp, I’m going to spend the hour shooting my shot. Even if it’s lower likelihood.
For random company XYZ, I expect humans to sink as much of their time into the process as I do.
I've worked at places that would send out 100. People would spend their weekends working on it and we often wouldn't even look at the submissions.
You're right the time commitment wasn't equal. Early on I spent much more time than the candidates designing and analyzing the test. Afterward, their 20 minutes would usually take me <5min (often <1min for obvious failures and obvious passes, the average brought up due to time analyzing edge cases).
I did read every submission though. It wasn't wasted time for candidates.
- It was designed to be fast to complete (20min max -- not a huge imposition if being hired is likely, obviously very expensive if you're taking one for every job posting).
- I only gave them out after a resume screen. If you had a 0% chance then I didn't waste your time. If you had enough other proof of abilities then I skipped the take-home.
- Candidates were told that it was designed to be fast and that if they couldn't complete it quickly they were unlikely to be successful interviewing either. They still had the option to spend a lot of time if they thought my assessment of the situation was wrong, but part of the point was to allow candidates to gauge their own abilities and not waste their time interviewing without a chance of being hired.
- I did a lot of work behind the scenes calibrating and re-writing the questions individually and as a whole so that the test score correlated very well with interview performance (most interviews administered by not-me, removing a form of bias that's easy to creep in there).