My last job search wasn't typical (I thought of starting a consultancy, so didn't search for a job per se), but one before that showed to me that it's exactly what people do. A company that was looking for a guy to write mobile SDKs asked me to build an Android app with embedded webview and animate html stuff with JS based on gyro movement. A slot game company asked me to build a minimalistic slot game from scratch (thanks to them, I have a clean minimalistic and complete game on github now). Another game company asked me to build a small endless shooter — and I ended up being the only candidate who moved texture offsets instead of actually moving game objects to infinity.
Each of those tasks took from 4 to 8 hours, and 2 of those got me an offer. And of course, I used the same method when I consulted others on hiring decisions or hired people myself. So, based on personal experience, companies do that, and it works.
For the the gigs that asked you to do these problems: did you also get interviewed with a standard programming interview? If you had to guess, between 1% and 100%, how much of the weight of your technical evaluation do you feel was on the sample challenges you did at home, versus Q&A and whiteboard coding on-site (or on the phone)?
My issue with this approach is fourfold:
1. Most companies have no idea how to structure a problem that is both informative to them and also not abusive to the candidates time.
2. Companies generally do this right after the recruiter phone screen, which most likely doesn't give the candidate enough information to decide if the next steps are worth their time.
3. Most companies still do a whole suite of normal tech screens after you work on a take home problem.
4. If you're actively looking, getting a bunch of these over a short period of time is likely. I know during my full time search, more than 50% of companies had a take home test right after the recruiter screen. Most of these were 4-8 hours of work each, due within the week.
A lot of startups structure it more like hazing or a barrier to entry than an evaluation criteria. I have some fun (read: horrifying) anecdotes from my recent search that illustrate the problems above, but I don't think any of my points are surprising.
A nice alternative would have been to simply have one or two projects completed that are straightforward to evaluate and walk companies through them, letting them ask me questions.
Does the work sample test offset all or most of the unstructured tech evaluation the company would otherwise do?
If it does, that means they have a work-sample test rubric they believe in, and they're hiring confidently. If it doesn't, they don't believe in what they're doing, and the programming challenges they're assigning can thus reasonably be seen as more hoop-jumping.
In the latter case, it's up to you to decide whether a prospective job is worth extra hoop-jumping. Some teams are worth it!
For example one company gave a problem with five parts, with the final part being solve longest path on a bipartite weighted graph (which is quite a hard and time consuming problem). After that, the next step was a phone technical screen, then an on-site with 4-5 more interviews, most being white-boarding. It was basically hazing instead of an evaluation criteria.
An alternative is my last job, which had a take home test that took about 6 hours, but that was the whole technical part of the process. Being on the other side reviewing them, the problem absolutely gave enough information.
I totally get there's a right way to do it, but like most interviewing trends, companies seem to just be adding this as a step instead of revamping their process.
Anyway, my larger point is that what I've been seeing interviewing is that these tests are becoming much more common at US startups without companies removing/reducing the rest of their technical evaluation process, nor really structuring the problems to be a good signal.
In an ideal world where companies do take home tests right, I think its a great solution. But what I've been seeing more often than not doesn't support that, making it hard to support.
I'm really curious what you've been seeing at Starfighter. Are partnering companies still going on to do a full technical interview? Or does Starfighter largely replace their normal technical evaluation?
Ignoring the fun of the challenges themselves (which probably isn't entirely fair), the latter makes it very compelling for a candidate. The former does not.
I'm actually fine with that! We make no pretense of having designed a screening process that is appropriate for every job. What I'm less fine with is the fact that the norm, even for abbreviated tech-outs, is 7-8 hours of on-site whiteboard interview.
Maybe this is an US thing. I changed jobs last year and nearly every single recruitment process involved exactly that, after a remote interview them giving me a small project worth 2-5 hours of work and then going over what I produced one week later. This was for companies in central and northern Europe.
There may exists engineers obviously awesome enough for that to be feasible, and great for them (but they are probably also obviously awesome enough to not have to go through coding tests, so it's a moot point) and even if I could perhaps (judging by certain recruiter-emails) scrape by as one of them, I certainly couldn't five years ago.
Back then, if I'd had to be good enough on paper alone to warrant not just being put through the recruitment process, but to be paid for it was well, I am not convinced I'd have been considered (and yes, I did a take-home test, and I aced it and it made up for my near-total lack of on-paper qualities).
Of course, companies shouldn't waste their applicants time with needlessly extensive tests, but there certainly exists no 'principle' by which you have a claim to be reimbursed for spending a few hours on an application.
I think that the same rules should apply for programming jobs. Just look at my resume and my publicly available code (that is linked from my resume). If you decide I am good enough to interview me, pay a reasonable hourly rate. This would stop the abusive practice of giving big problems to solve in our free time.
This is a sound-bite, not something that holds up to scrutiny.
If you decide not to interview with possibly great places based on this, you're essentially saying that all future expected gain isn't worth some set hourly rate for 2-8 hours of your time.
Also, what's your free time worth? For me, sometimes it's worthless, and sometimes I feel like I'd sell my soul for 10 minutes to myself.
The last "shouldn't take you more than two hours" exercise that I was given involved jumping through hoops to get set up on Instagrams API, only to discover that I was in a sandboxed mode, and needed my account approved before i could get anything more than metadata out). I didn't even get started coding, so I am not sure what exactly the test was meant to achieve.
If all it required was an hour phone call and a small project, I'd totally do it for the right company.
I'm too well-paid and busy to do interviews.
Have every candidate work on the same problems, and
let them do it from home
Regardless of whether you agree with me about that, I think we can all stipulate that if work-sample technical evaluations work (and: they do), most of what companies try to evaluate in on-site interviews is stupid. No part of working effectively with a team requires timed recall of how to implement a stable quicksort, or reversing a doubly linked list at a whiteboard.
At the very least, using work sample tests allows you to build an on-site interview process that honestly engages with "team fit" (or whatever your X-factor is).
My guess, though, is that when more teams adopt work samples and then go through the motions of trying to design a pure team-fit interview, they're going to realize --- once they don't have "implement Bellman-Ford on this whiteboard" to fall back on --- how unequipped they always were to evaluate team-fit in the first place.
The ridiculous whiteboard coding of puzzles probably stems from laziness or inability to implement what you're suggesting. I know that personally, if someone asked me to developed a work sample evaluation for my job, it would take me many hours to come up with something, and frankly I even once complete we would have no way of knowing if it is actually a good predictor of whether or not someone is the right employee.
I'm religious about this topic for a reason. :)
No, I attribute the turnover to the recruiting mechanism. We found great people who were sorely mispriced by the market, and we took advantage of that to create a win-win scenario: people without the resume to get a similar job at a competing firm got an extremely impactful resume bullet and a good-paying job, and we got people who genuinely wanted to be on our team and weren't applying as a once-a-year lateral job-hopping gamble.
(I have no problem with people job-hopping, by the way, but every employer is trying to minimize their exposure to that.)
Really my comment was just to help demystify why this sort of thing doesn't happen in many companies based on my own observations.
The candidate shares their screen with you, so you can watch as they solve the problem in their own dev environment. You can understand how they approach problems (quick and dirty, slow and methodical, lots of rewriting, etc), and you can ask questions at the end. You get a good sense for how they work as an engineer even before having to bring them onsite.
This seems to resolve the time asymmetry of take-home tests as well -- the interviewer spends as much time watching as the candidate spends working.
The only downside I can see is that you have to design your problems to take about an hour instead of the 2-5 hours you could imagine for a take-home test. But, you can break them up into multiple rounds, and give additional exercises to the candidates who do well on the first one, for no more total time cost than a collection of onsite interviews.
For what it's worth, I've done this at my startup and hired a great developer, and got very positive feedback about the process from the candidates I didn't end up hiring.
My ideal interview format would involve a work sample test, followed by a code review of said work sample, and also a reverse code review (candidate is given some code to review). As an added bonus, all of this can easily be done over email without wasting too much of anyone's time, especially compared to an all day in-person interview with lots of whiteboard time.
1) HR (presumably taking direction from in house counsel) takes the position that it will be difficult to distinguish between a work sample test and an IQ test
2) We are a large organization with very broad roles. The Performance Team hiring for a Senior Software Engineer will probably want a different work sample than the Analytics Team...
Let's not dignify that argument. I absolutely believe you that your company has allowed HR to sabotage your hiring process, and that sucks. But let's not pretend HR is right to do it.
There's probably a solid argument for giving every candidate for the same role the same work sample test; otherwise you might naturally want to randomize some parameters or choose from a bank (if you think "cheating" is likely).
If you or anyone you know is interested I'm a software developer and spend a lot of my time on cloud architectures/containers/multi tenancy optimizations but just generally enjoy solving business problems with technology. In a past life I cofounded a startup that didn't turn out to be viable after going through an incubator (not YC). I have a resume, LinkedIn, Github etc.
Oh, what a terrible tragedy, to give candidates consistent evaluations that are actually comparable between each other... /s
I get the point you're presenting, but at the end of the day it still boils down to "we can't use tests because then we'd have to be held accountable for objectivity, we prefer instead a system that lacks any kind of rigor such that any result may not be effectively challenged, officially or otherwise".
There isn't enough feedback when someone has a bad idea on hiring. People will very quickly tell you if your code sucks though (this is a good thing), which might be the kind of feedback needed to get hiring changed.
I think it's pretty dumb to do that, though, because most software developers can't take a month off of work from their current job to see if they're a fit for your company, and none of them will put their health insurance in jeopardy to do that.
I also think it's pretty silly to design a process that deliberately demands a month of time to make a decision that could be made effectively in a week.
Its not usually that hard for a contractor to take the time off; they're on their own time already.
we were going to get together in a room anyways.
* Our in-person interviews were shorter than typical in-person interviews because they weren't tasked with fully teching candidates out.
* Because they didn't try to tech candidates out, they weren't as stressful, and so were less draining and unpleasant.
* Because our in-person interviews were entirely scripted (the interviewers had very little latitude with what they could ask and how they could ask it --- which they hated, by the way), they were easy for everyone to deliver.
* Before candidates arrived to their interview, they already knew (because we told them!) that they were likely to receive an offer from us based on their performance on the work-sample tests.
I understand why people resist the idea of coding challenges, given:
* They're not going to get feedback for weeks, months, or maybe ever
* The challenge is going to be graded pass-fail, or whimsically, without rigor or consistency
* They're going to have to do the exact same grueling bullshit draining dehumanizing programmer interview anyways
* They're not going to see the challenges coming, or, for that matter, know exactly what happens next after the challenges are done
That process is super common, even at companies that ostensibly do challenges. It sucks. I agree, no company can really get away with this in the long term.
But those are also chickenshit problems. Switching from unstructured interviews to work-samples is hard: you have to change your mindset on how to qualify candidates, and there's a leap of faith involved. But getting candidates feedback, telling them what to expect with your process, keeping a schedule, and having processes in place to be consistent aren't hard problems. They are table-stakes common sense business execution, and if your team can't handle that right now, your team is mismanaged.
We had a grading rubric for the challenge that was based on the kinds of things people found in the application. We evaluated performance on the challenge on a 1-5 score; depending on how you did on other challenges, a 3 would keep you in the game, and a 4-5 probably assured an in-person interview.
Could I have a copy of that script?
With standardized interviews and challenges, our whole hiring pipeline became entirely mechanical. We could send our a challenge and its instructions in 2 minutes, and evaluate the response the next day (or week, or whatever) later in less than 5 minutes. 7 minutes, to produce a technical evaluation that crushed a competing battery of tech interviews by several senior staff members that took 4-5 hours to do.
Back when things were simpler, we had a multiple guess questionnaire that did the similar. For onsites, we then looked for evidence of rigor and thoroughness, the ability to estimate how much communication was required, had they deployed real systems?
You understand that it sounds like you "fixed" hiring, and so we're all sorta skeptical, right? If you could bottle that, there's your billion.
Announcing Starfighter https://news.ycombinator.com/item?id=9173939
The biggest complaint we'd been getting from candidates before work samples was that our process took too long. Our goal became to get the entire process done under two weeks (from its worst point, a month or two earlier, of 1.5 months, when someone finally posted an anonymous complaint to Glassdoor about us).
After iterating a couple times, we could reliably go from first contact to hire/no hire, with a motivated candidates (ie: one who would work with us to schedule the interview process aggressively) inside of a single week.
Also, see: https://news.ycombinator.com/item?id=11328090
The process I'm advocating for is far better for candidates.
What do you do to ameliorate this concern?
I am spending a lot of time these days talking to people interviewing in the valley, and what I'm seeing is that the norm candidates are subjected to is 7+ hours of 6-7 on-site interviews. Candidates have to go through all the interviews, even if the first interview has effectively ejected them from the process.
Compared to that horrid process, I don't believe I have to justify anything about my process.
But, if you read downthread, you'll see that we in fact did a lot to ameliorate the (bogus, I think) concern that we were incentivized to soak up hopeless effort from lots of people.
Think of the design world:
Potential client "A" asks multiple desperate artists to work on spec in hopes that they will get the commission.
Potential client "B" call you up and talks to you, sends you some napkin sketches and generally engages with you for an hour before asking you to do a design.
Both clients want you to do a design (test) but one clearly doesn't have any skin in the game. Assuming you're a decent designer, which client do you respond to?
Companies that don't hire using objective at-home tests hire instead with grueling on-site interviews that knock out an entire business day (most of them last a whole business day, some recent interviews peers have gone on have taken multiple business days, and all of them at the very least kill the day for anything else). The latter is obviously worse than the former; I don't even see how any other argument could be colorable.
What I'm beginning to think is that this kind of pushback comes mostly from developers who are well-connected, and so they never experience the grind that less well-connected but equally-capable developers do with interviews.
Yes: if you're at a point in your career where you can get a job in any of 6-7 different companies just by raising your hand, saying "I'm available", and having a 30 minute conversation with the VP/Engineering who you worked with 2 jobs ago, everything is a waste of time. I am not here to tell you that you should make things harder for yourself.
The problem is in the hands of TrendCo, they can be used crudely to find the trendy hire who has the exact same ideas about engineering as the hiring person. Because giving the test costs TrendCo nothing, they are happy to throw people at it until they randomly find someone who is exactly what they are looking for.
Anecdotally, I and 2 people I later befriended applied for the same role. We each spent 4 hours on the task, all 3 of us strong coders who made a solution that would be suitable for any startup, all 3 taking different approaches based on our styles. We were all rejected, because none of the 3 of us hit upon the exact approach the company was hoping for, but didn't ask for. There was no opportunity to ask for this feedback either. And it took them nearly a month to bother looking at my code (even after I called).
Since then, I won't take these kind of tests unless I have reason to believe they are being given in good faith as a way to determine if a programmer is capable, not as a way to find their perfect ideal of a programmer.
I have the strong impression that a typical SFBA tech hiring manager thinks it's crazy to give candidates cheat sheets for interviews, and that helping candidates with technical evaluations decreases rigor. THE EXACT OPPOSITE THING IS TRUE. Evaluating technical ability under unrealistic pressure and with unrealistic restrictions confounds the results of the evaluation.
Of course, you have to design tests that are valid predictors for candidates who have received tips and assistance and practice and resources. That sounds hard! But it isn't. The best predictor for a candidate's ability to do the kind of work you really do is, simply, the kind of work you really do. Not problems from a book, not trivia questions, not one algorithm poorly implemented on a shared coding site and a Skype call, not whatever bug is at the top of the bug tracker today, but a sample assignment given to a real member of your team, the same one, for every candidate, graded the same way.
It is amazing to me that almost nobody does this, but: almost nobody does this. They have programmers write code on a whiteboard, or on some whiteboard-coding site; they have them do programming puzzles ("solve Towers of Hanoi non-recursively"), they have them talk about code, or, more likely, computer science trivia. They'll have them "pair off" with one of their own programmers and "fix a bug". They'll have them work as a 1099 for a month to see if they're any good.
In reality, most companies are trying to hire on trendiness; they want people from the right schools and/or the right cohort of companies. They're aggressively courting friends and colleagues of their existing team, and the special people get very different interviews than everyone else. The actual technical evaluation is mostly a facade.