I was filled with anxiety working under the time pressure to understand a document of code which I had very little context of, and no previous relationship with the reviewer so I kept thinking to myself whether or not the question that came into my mind was stupid obvious.
In contrast, when I had take-home tests, I scored exceptionally high on all of them, leading to offers. I recognize that a lot of people here hate them because of the time involved and it's akin to spec work, but IMO it felt much more like my usual day-to-day life as a developer.
One of the take home tests had existing code to work from but again, working at my own pace and having access to online resources to read docs was much less anxiety provoking.
You say:
> One issue was the syntax was so weird compared to the type of code I'm used to writing and reading from others in library code that it was hard for me to comprehend the logic without the benefit of looking it up.
From the point of view of an interviewer, that's feedback I'd be interested in. One problem I've seen in some of our code is an engineer using a completely valid but very obscure method of doing something, and then the rest of us having difficulty maintaining it. I would want to hear, "this looks hard to maintain." But I understand that in an interview, it can be difficult to assess what the interviewer wants to hear and that can be nerve-racking. I personally think take-homes are a reasonable compromise, but not everyone does.
1. It fosters meaningful conversations about code that is not dependent on the candidate solving a toy problem.
2. As you said, it tests a candidate's ability to read and understand code.
3. It's actually representative of the type of work you _will_ do at any given company, i.e. code review.
The major downside is if you give the candidate code in a language they're totally unfamiliar with then that candidate will likely bomb. I think this can be mitigated by sticking with mainstream languages like C/Java/Python, or domain specific languages where not-knowing would be a deal breaker anyway, i.e. must know Swift for iOS dev.
It also works even if the person doesn't know the language because you'll get to see them mapping this new language to something they already know and it can give a good understanding of how well they think of things beyond just the code in front of them.
I have create a git repo specified for the question, with some code working in way, but buggy, such as logical bugs or race conditions...etc. with a lot of not business relevant assumptions. I usually told the candidate what this code supposed to do, and ask him to review the code first, then tell me what was his impression on the code base, and what were the problems, and how he can fix it. By doing this, I can judge the candidate if they are good at general coding, debugging and and comfortable to work on projects which they are new to, as well as their mindset on writing maintainable software. A good candidate will not just fixing the obvious issue, but propose some architectural changes, then I will ask them what are the strategy to do so as the project has existing dependents, this can give me signal on how the candidate coordinate with other teams. Then I will ask him some more questions related new features, and scaling..etc. to measure his design, communication skills and more.
It never failed me for senior level positions hiring, that either the candidates were hired and proven to be a great match, or they got a better offer elsewhere.
Pick some interesting problems from there, and without asking the candidate to actually code a solution, discuss the problems with the candidate. Ask for ideas on how to approach the problem, and suggest some yourself if at any time they appear to be getting stuck.
After they, possibly with guidance from you, have reached an approach that if correctly implemented would solve the problem, including meeting time and space requirements, discuss what issues might arise in actually implementing it, such as edge cases that might require special handling. This would be a good place to talk about how they would go about testing an implementation.
Finally, go to the discussion on LeetCode for that problem. There you will find various solutions posted by others. What I have found looking at those is that a lot of them don't meet the time/space requirements, or miss edge cases, or sometimes are just wrong. Go over a few of these with the candidate and have them criticize them for you.
You can save some effort though and just hire anyone who passed Chinese or Korean grueling school system with emphasis on standardized tests. LC-based interviews follow the same tradition.
One strategy that's been successful for me is skipping the technical screen and jumping to a narrowly-scoped paid assignment. If they do well, I give them a larger paid assignment, and we work our way up to regular work together. If they do poorly, I pay them for their time but tell them it's not a match.
I'll add the caveat that I've only used this to hire junior to mid-level developers for part-time or short-term work. I've never tested it on a full-time hire, and I imagine that would be more difficult.
I've never written in detail about hiring developers, but I've shared similar strategies for hiring writers,[0] illustrators,[1] and editors.[2]
[0] https://mtlynch.io/hiring-content-writers/
It really explains a lot about the C-level people and their lack of engagement with work (for my company at least!); they don't seem to be doing their full jobs as, well, they aren't. I really need to figure out how to get on this carousel.
Kudos to you. You have worked out one of the fairest ways to gauge someone. May your tribe increase!
It always seems strange to me that people won't bat an eye at taking a day (or more) off work to come on-site for technical interviews, yet the idea of giving someone a 6-hour project do at home, at their leisure, is somehow a bridge too far.
In real-world hiring experience: I gave candidates a choice between on-site interviews and take-home projects for a while. Zero people chose on-site interviews when given the option, so I stopped offering it.
The bottom line is that interviewing requires time. If someone isn't willing to lift a finger or allocate a few hours to an interview, how are we supposed to gauge their competency? The "just trust me" model doesn't work at all.
One of the key differences is the company has "skin in the game" at an interview, but not in the 6-hour project.
When you interview, you know the company has included you in a shortlist and they will put time into getting to know you. For the time put in, you have a chance to pitch yourself, to at least get across your general abilities and personality, and some people will pay some attention. Also you can learn more about the company and the people you might work with.
When you get a take-home project, it's like the way spam is cheap: It takes almost no effort for the company to send the same project to a lot more candidates at low cost.
You do the work, but there's no guarantee the company will really assess it. Sometimes the company drops the ball and just "forgets" the application, or discards it the same way résumés are filtered: By skimming though 6-hour project submissions and casually throwing some of them away. It's not uncommon to not hear back anything from a company after sending the result of working on their project.
Some candidates will enjoy doing a project more than an interview, but that's not the only consideration in determining their value to the candidate.
Personal experience: I have never had an interview last more than an hour. I have also been hired twice on a "just trust me" model and it worked reasonably well.
Replacing a day of interviews with a 6 hour project sounds fair. That isn't how most companies use projects.
I can understand not wanting to do that if you're a full-time employee, but if you're a freelance developer actively looking for jobs, why would a paid project be worse than an unpaid phone screen?
Why would this exclude people with family obligations? When I say "narrowly-acoped project," I mean 5-10 hours of work. I've never gotten an offer for a dev role where the application interview process took less than 5 hours, and those were unpaid hours.
In other words, some visa holders in the US are very much allowed to work, but not contract work.
5-10 hours would be similar to a typical interview loop yes, and if that was the entirety of it then it's not more time consuming than the typical engineering interview.
The alternative is to have them take time off of their dayjob and travel on-site for technical interviews.
In the real world, candidates are greatly appreciative of take-home assignments that let them work in the comfort of their own home, on their own equipment, at times that work best for them.
I feel like this is the solution that values candidate time the most. They don't have to do an interview, and I'm paying them for their time immediately.
Is it really worth it to lose a day in your weekend for a couple hundred bucks and the hope of getting yet another assignment or a job?
And, if we're being really honest here, there's a ton of informal backchannel talking that goes on, so your interview "performance" regardless of medium is affected by that at many places. I think if you aren't actively working to make your interview process objective, it's being strongly affected by many biases and informal relationships.
That's all to say that on the balance, taking up "some" or "all" of a candidates time isn't that interesting of thing to think about because the process is so broken in so many other ways and this one doesn't have a huge impact.
I can't get that from a take home assignment.
I have 20 years of experience in X but prove it? Why's the burden on me to prove what I've already told you? Are you saying I'm a liar? If my date of birth is on my cv do you expect a birth certificate to validate that information too?
Anyway, that's always been my philosophy - hire on facts and fit, fire quickly if anything was wrong - and it's been pretty smooth. The great thing that it does is it prevents testing for futile thing that you will never use in your position and that are just elitist criteria or vanity wishes.
The reason this philosophy is misguided is not that candidates flat out lie all the time (though that happens too), but more that resumes are crafted to inflate, exaggerate and obfuscate to make the candidate seem more qualified than they might actually be for the position upon closer scrutiny. I say I have 20 years of experience in X, but 4 of those years were just hobby projects in high school, 4 studying in university, another 5 in entry-level positions in a semi-related industry, etc etc. It would be a huge waste of time in obviously unnecessary hirings+firings to blindly trust stated experience on a resume without verifying its accuracy first.
Engineers who are 100% honest, overly humble, and fortunate enough to have only worked at top-quality companies tend to not understand the need for verifying the skills that people put on their resume.
In the real world, some of the best candidates have the worst resumes. Some of the worst candidates have the best resumes. And perhaps most dangerously, some of the worst hires tend to be the best at lying and exaggerating because they've used their charisma to get them through life.
Making bad hires is a very costly mistake for the team. Not only do you lose out on months of team-building and project progress, but it's bad for morale to see people hired and fired quickly. In reality, bad hires are going to stick around for 6-12 months or more at any company of moderate size company.
Also, not all experience is created equal. It's shockingly common for applicants to claim 5+ years of experience in something, only to discover that they only touched that topic a couple times per year for 5 straight years, for example.
US hiring practices expect to see something that validates your birthdate, so kind of.
Fast firing can work, but none of the places I've worked were prepared to do it. Similar to what a sibling commenter posted, years of experience is something, but what you got out of it is another. There are plenty of people who have spent a lot of time doing something and not gotten much experience. In my mind, a good interview process figures out where the candidate actually is with regards to experience and capability, if they fit with the company, if the company fits with them, with as minimal a time commitment for both sides as possible.
It's not great for either party if someone is hired and fired in a month.
Indeed. People also underestimate just how quickly you can learn in the right environment.
Yes, and the way that I know this is by performing the tests you're arguing against. I've interviewed plenty of people that could converse fluently in buzzwords, but could not write code. In the actual job, it's necessary to be able to write code.
Lots of jobs aren't all that demanding.
Do they quiz them like they did to the residents on Scrubs? Do they pull up a list of symptoms and ask what they would test/try next? Do they show them x-rays and ask them to spot the coronavirus?
Seriously? Or do they look at your degree and experience and conclude that if you worked at General Hospital in Atlanta for 5 years you can work at Metro Hospital in Philadelphia?
I honestly don't know how other professional industries hire industry-hires (not college grads/interns) but I only hear about quizzes and riddles and online tests and shit like that in programming.
Hell, do they do the same shit for electrical engineers? Chemical engineers?
I really want to know if they do and if not, how is it Boeing can hire a EE without a whiteboard puzzle but can't hire a CE without that bullshit?
I think it would be a shame to bring this sort of thing to software development, but it would allow us to get rid of some of the hoops people have to jump through to ensure they can actually do the job being hired for.
I'm not that opposed in general to whiteboarding if you feel like you can't assess the candidate. However I do feel that it sometimes feels like credential and experience similar to what you describe in the medical field: if a doctor had 5 years at an inner city hospital, I don't think they would doubt his credentials to the point the want to make him put cell structures on a whiteboard?
Do you really want the same for programmers? "only people with Ms or PhD from accredited universities can apply, and diploma is invalidated on some mistakes"
In some industries an engineer might need to be licensed. That requires passing multiple tough exams and working under another professional engineer to get their license.
Physicians are similar in that you have board certification and residency which provides a baseline level of implied compatency. As I understand it, your residency can essentially set your career. You won't be able to change speciality, and moving up in terms of institution is difficult.
In practice this might be "you can only get hired at faang with faang on your resume", so your options if you don't get it right out of college are to do masters or hope to get a job at Amazon and possibly pivot that into a position at a different firm.
I did whiteboards for Architect positions, but once again, I just asked the candidates to draw architectures to have a friendly discussion.
This is the fundamental part that is broken in my experience.
They are often challenges/coding puzzles, when rather white board exercises should focus on collaboration, and solving problems with code.
Having a "gotcha" that shows rudimentary CS understanding is fine (such as understanding why the obvious O(N^2) is a bad solution). But having a gotcha that requires some crazy leap of faith guess (or having seen the problem before) is not.
I admit, my fondness for a candidate increases with the level of complexity of coding problem they can solve; however needing hints on the harder problems doesn't mean they are not qualified.
I'd be extremely surprised if Jeff Dean (at any point in his employment) would fail a SWE interview loop at Google. The Google interview is heavily based on data structures and algorithms, and Jeff Dean likely knows them cold.
Sure there's always a degree of luck, and Google errs on the side of false negatives, but there are certain people that are so high above the hiring bar that luck becomes too tiny a factor to be relevant.
That's pretty much what Leetcode is. When I interviewed at Google I was asked tree traversal and hashmap design questions that were very similar to leetcode questions.
Questions are mostly about technologies that hiring team needs, the more the better, and the deeper the better. I never had a problem getting the extra experience not found on previous workplace from my pet projects and blogs before moving to US.
Funnily, I've seen people who moved back from SF Bay Area, and struggled to find a job there, despite being very good at whiteboard and CS problems. Turned out eastern european companies don't care about CS problems really.
This is absolutely why whiteboarding exists, and persists.
I have interviewed a lot of people at this point in my career, and it's just not possible to tell if candidates are good by looking at their credentials or job histories.
I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions. I'm not talking about Leetcode-style brain teasers; I'm talking about being unable to construct a working for loop and do basic array operations in a language of their choice.
I'm going to continue to do whiteboarding for candidates for this reason.
That honestly sounds to me like you found some people who lied like crazy about their credentials and job histories.
I work somewhere in the middle of biology/neuroscience and ML. Interviews for essentially the same job (e.g., building brain-inspired robots) are hilariously different depending on how the org views itself.
In a "science" organization, the interviews mostly involved talking about what I've done in the past and how I'd approach new issues (technical and behavioral). There were certainly a few probing questions about things that seemed designed to check up on claims, but my experience seemed mostly taken at face value. A coding test, for example, was skipped because an interviewer had already seen some of my code and thought it was fine.
"Tech" interviewers seemed to want everything "proved" on-site during an all-day slog. I reversed strings and traversed graphs on a whiteboard. Someone quizzed me on facts and figures about the human eye (which they were reading out of the same book I learned them from). To me, this seems to test immediate recall and stamina/stress tolerance more than anything actually related to my R&D skills.
I think something more akin to a thesis defense would be better than what we're doing now. Prior to the interview have the candidate write up a description of a project they worked on and particularly liked along with a set of slides. Have the candidate send this presentation to the main interviewer who will distribute it to all of the interviewers. The interview then becomes a presentation where interviewers can ask questions about the presentation. I've seen this done in industry, but it was only when we interviewed folks who were PhDs coming from academia - I don't see why we couldn't do this for all kinds of candidates, not just those coming from academia.
1) What did you, personally, work on previously that's relevant to this role? This usually means pretty in-depth discussions about previous roles and projects, accomplishments, tasks, skills gained, etc.
2) How do you approach tasks and problems at work? This is usually a combination of asking about why the candidate did things a certain way previously and how they would approach theoretical situations in the future. The goal is to uncover the mindset the candidate uses to actually get stuff done in a changing business environment.
3) How do you interact with teammates and managers? This is the concrete version of "culture fit" and includes asking about various interpersonal situations likely to have been encountered in the past as well as discussions about how the candidate would handle quirks of the office environment at the new company (ex. some companies love people to make decisions quickly on their own while others hate it and mandate team consensus).
4) Do you have the base-level skills needed for the role? This is often far more than the technical aspect. After some basic questions to establish that the candidate knows the basic processes, software tools, artifacts, and terminology, it's much more important to establish if they have the chops to actually be successful rather than a "task completed" robot. Can the candidate communicate ideas and opinions clearly? Did they previously work in a collaborative way rather than an adversarial way with teammates? Can they present findings and issues to management effectively? Have they previously worked with supporting teams and vendors outside their main role to solve problems? Etc.
In most jobs the technical skill part is a small piece of what the person will actually be doing so the interview is a means to collaboratively establish if the role is a fit for the candidate as a whole and vice versa. If I only looked at technical skill and immediately disqualified people for failing concocted and unrealistic tests, I'd end up with mostly shitty candidates who aren't actually very good at the wider job.
As have already been reiterated in each and everyone of these threads (about leetcode/hackerrank/whiteboard coding) this is what the companies seem to think is the most proven, and produces most consistent results. But I would love to understand how other industries does this as well; does a propulsion engineer start to draw up a de Laval nozzle and calculate the exhaust gas velocity on a whiteboard? (again genuine question)
Other industries have higher and more consistent standards, and certification processes. The software industry is the wild west, and that is partly why it has succeeded and why so many self-taught programmers have found success in it. The other side of the coin is that the quality of candidates varies more widely than in other industries.
I think the candidate should be given a choice - whether they want to solve one of these tricky puzzles or work on a domain problem.
Imagine you're a team of 2 top performers. Your workload is increasing, so your manager gets approval to hire a 3rd person so you can take on more work. Unfortunately, your manager is fooled by a charismatic person who greatly exaggerated their experience on their resume, to the point that they work about 1/4 as fast as the rest of the team and tend to code by trial-and-error.
Suddenly, the expectations for your team's output have increased by 50%, but the 3rd person can only really contribute another 10-20% at most. By extension, your job just got harder.
Worse yet, the average talent of the team has now gone down significantly. You spend more time fixing bugs and chasing bad work. You're also faced with the fact that you're probably getting paid the same as the over-leveled person next to you. This breeds resentment, wastes time, and ultimately leads to good people quitting for other jobs.
Alternatively, imagine if the company hired quickly and fired quickly. Every time a new person showed up, you had no idea if they were actually good or if they were just rushed through the interview process. You also don't know if they're going to be around after the 3 month probationary period, so why bother building a relationship? High turnover like this tends to destroy morale.
If someone has worked at 6-8 places over 15/20 years covering a range of skills required I don't see the risk.
You tell them what the job is like, what is expected of them, and what the best way to work with you is, then you you ask if that's what they want, and how much it's going to take to get them in the door. That's it.
I wouldn't ask an accountant what the Corrodium 3 Deduction is; I'd tell them I need them to file my taxes, that I need them to own the process and chase whoever they need (even if that's me).
> I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.
I think you're probably right, but if someone lies to you and says they can do a job that they clearly can't, you should fire them.
This should be unusual because getting a new job can be stressful, and nobody wants to put themselves into a job they can't do, so it's rare that you even have to consider this -- but it does happen, and unless you're a sociopath, it's going to make you shy about ever having to go through it again, because you're not going to want to fire anybody. It sucks for you, and it sucks for them, and it sucks for the team who see the revolving door, but nobody benefits from the wrong fit in the long run, so you just need to suck it up and do it.
So be firm, and if it turns out you're firing a lot of people straight in the door, look inwards: you're probably not being honest with them (or perhaps yourself) what the job actually is. Figuring that out instead of some new programming puzzle to torture grads with is going to pay dividends.
You can have code challenges that aren’t HackerRank or Leetcode.
Using problems from those sites is a signal the employer or team is lazy.
He's previously said that if he could only decide based on one or the other, he'd hire the candidate referred by a trusted source and skip the interview. In practice, this choice never comes up, but you can weight referrals much more strongly than interviews.
Genuine question here: what's the better way?
I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job.
On the other hand, take home challenges eat up a lot of time and credentialist filters like going to a name brand school (or even having the right major) filter me out.
My go-to for when I'm in a hiring position again would be recommendations from my personal network, but that has its own flaws and doesn't scale very well.
I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.