For example, a good portion of doctors absolutely hated using checklists. Yet, when pressed, readily admitted that it prevents simple mistakes and that they would prefer to have them rather than not to. Another is that entries that address more human concerns, e.g. "Have everyone introduce themselves", have a place on good checklists.
[0] http://www.amazon.com/Checklist-Manifesto-How-Things-Right/d...
At a high level, this is all about Deming ( http://en.wikipedia.org/wiki/W._Edwards_Deming ) and TQM concepts -- if you want to achieve a high-quality output, measure the things that matter, and understand the variation present in the system. Once you have a stable system with good data achieved by good methods, you may then begin improving it. Attempting to improve a complex system without knowledge results in unpredictable changes—we call that tampering. Simple but beautiful.
So, in essence, this is an extremely natural and correct application of quality management principles to the hiring process. Stellar.
discussion: https://news.ycombinator.com/item?id=8830903
In your post you say, "At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began." You seem to continue on from that point. What happens before first-call?
What surprised me about recruiting was how much more important qualification is than outreach. Without good qualification, it almost doesn't matter how good your outreach is, because you're filtering for the same highly visible easily accessible candidates every other firm is looking for. The best-priced talent is buried. What you need is ground-penetrating radar. Clever outreach schemes helped us, but what really let us make our best hires was a qualification process that allowed us to confidently ignore resumes.
We didn't just maintain our team's level of quality by ignoring resumes. We transcended it. We never could have run a resume-based process that would hire people who could use lattice math to break crypto. To hire those people, we needed to select for aptitude, not experience.
I haven't said it anywhere yet, but in case the subtext isn't clear: the person who can implement an attack for which edit/compile/debug takes 4-6 hours also does a pretty great job on every other facet of the software security job. There are aptitudes that are very good proxies for other aptitudes!
Maybe my premise of 100 candidates is incorrect based on how you did outreach?
How do we figure out which ones are serious? We talk to them, usually give them homework, and then give them work-sample tests. This is not particularly expensive for us, and gives us repeatable, relevant, apples-to-apples metrics on how qualified they are to do the work. This is, as Tom insists, surprisingly easy for us. (Not to say it's easy, but nothing about hiring is easy.)
Two questions for you.
What is your experience hiring fresh college grads? Is the process different? Since one of goals of the process seems to not arbitrarily adjust it for every candidate, but rather stay consistent, do you think it makes sense to adjust it for someone who just graduated. They might have been learning computer science type stuff, data structures, did a few group projects and have certainly been practicing the typical interview puzzles, fizbuzz and O(n) times of the all data structures.
How do you account (or do you account at all) for culture fit. That is a fuzzy and slightly dangerous area. Is there a score in that regard? There are some people who are brilliant but they do not work well in a team, or rather would not well in one particular team. This is course full of danger as "culture" fit often becomes an opaque discriminator for racial, sexual and other stereotypes. "He's just not a good culture fit" is such an overload euphemisms that I feel kind of sleazy just saying.
The lede I buried in this post is that we hired more-or-less resume-blind. I had final call on every hire and went through some effort not to look at resumes. On our first-calls, I'd ask some questions that would give me background hints about candidates, but that was primarily to tune my spiel (I didn't want to explain blind SQL injection to someone who'd spent 3 years as a pentester already, for instance).
If I had it to do over again at Matasano, I'd have done it formally resume-blind. Only Dina would have had access to resumes, and interviewers would be forbidden to ask resume questions. The process would have worked better if resumes were firewalled out completely.
There was no culture fit score; in fact, if you gave subjective feedback ("passion", "confidence", "work ethic") your feedback was likely to be discounted. We were proud of the culture diversity we managed to scrounge out of our candidate pool; to have team members with kids who needed to keep strict hours, and team members who'd roll in at 11 and still be there at 8 working on a pet problem. We had drinkers and non drinkers, college students and people with 15+ years of dev experience.
You could get booted from the process for being overtly an asshole (I, for instance, might not get hired), but that happened very rarely and with crystalline clarity.
Don't get me wrong, I understand and agree with your reasoning. I'm just curious how transparent it is to the other side.
So really "what did you study in college?" needs to come off the list of things you ask (at least as part of the filter).
on the reverse of that, i have sometimes avoided topics that the candidate was familiar with because i didnt see relevant experience on a resume. that was a major miss, and ive tried to avoid using resumes for shaping the interview topics since.. i see the resume now as more of a signal of what the person is interested in (since most of us carve huge swaths of our experiences out, and tailor them for the job we apply to anyways).
Good job coming up with something different. I hope it's really working out as well as you say.
Hmm, I once took it upon myself to do exactly that (mostly using non-blocking sockets, skipping hostname lookups, etc.), but what's so "tricky" about this question? Assuming the position involved network programming, I'd consider it a good way to test someone's networking knowledge. Is the "tricky" part knowing how routers often drop ICMP replies if they exceed some threshold rate?
That's nowhere near as bad as the interviewer who has the primary goal of demonstrating how smart he is (to the person he is interviewing), with the actual hiring process being secondary.
So, I'm not a fan of giving interviews. I wouldn't mind shadowing someone better than I, but my first inclination is that that would be a terrible idea since it would make candidates even more nervous.
This may not be an issue for you yet, but what about plagiarism? And can you expand on your hints about how you changed your pipeline at the front of the process? You don't interview every resume that comes your way, right?
I did a lot of calls where I knew a couple minutes in that we were unlikely to hire the person, but (a) I got surprised by outcomes enough not to shirk on those calls, and (b) those calls are a very small price to pay for finding buried talent.
1) People generally don't cheat. If you have a high percentage of cheaters its how you are filling your pipeline you need to address, not the filter.
2) What most people think of as "plagiarism" is actually very common in the real work of software developers. Very frequently you see/mimic other peoples work to solve problems. Instead of being freaked out about a skill that is central to the job, why not use it to evaluate the candidate. Did they "plagiarize" the right thing? Did they do it effectively. Did they do something backwards that a simple google search would have found a thing to copy?
3) If you are big enough for plagiarism to be a real problem and have addressed points 1 & 2, it is relatively easy to detect mechanically (and there is a surprising amount of research in the field as CS professors invariably write both an automated grader and then an automated cheater detector).
I was wondering if you could share some info on the salaries. Did you offer every candidate the same salary or adjust it (based on resume/previous salary/age/experience/...)? You mentioned 100% retention - so I assume no candidate ever left for another position with a higher salary?
Being a specialised consulting firm, I imagine Matasano could offer highly diverse, very interesting work long-term (because consulting), and higher-than-average salaries (because specialised), so maybe it wasn't a problem for you to offer higher salaries than the competition, or to retain candidates at equivalent salaries.
Great article. The prelude to the interview with books given to candidates is especially interesting. I'd love to see that reading list, and would further like to suggest posting it on your careers page. I would imagine that anyone who goes to the trouble of working through your suggested materials would be providing a very strong signal of quality and investment.
Also, what percentage of people who complete the work-sample task get an onsite interview?
I'm asking these questions because I think most software companies who filter out people based on resumes and phone screens do it mainly because there are too many applicants and they need to filter out many candidates at earlier stages. Do you think your hiring strategy resulted in more time spent on the process per successful candidate and per applicant?
As someone who already has a job, committing a large number of hours for the chance of an interview seems like a waste of time.
If every employer demands a 5-10 hour work sample test before they even talk to you, that really isn't a scalable solution from the viewpoint of a candidate. I could easily put in 100 hours into works-sample tests without getting a single decent interview, so now I just refuse them.
Everyone else is screening people based on resumes? Then, you can find good candidates by ignoring resumes, so you find good people with bad experience.
Nobody else hires people older than 40? Then you target people who are older than 40 but are still good workers.
Everyone wants a programmer with X experience but there's a shortage of people with X experience? Then hire good programmers without X experience, and give them a chance to learn X on the job.
Nobody else gives a work-sample test? Then giving a work-sample test is a benefit. You find candidates who are desperate and good enough to do the work-sample test. If everyone gives a work-sample test, then you get an advantage by NOT doing a work-sample test. (For example, a company with a work-sample test is now excluding me from their candidate pool, due the the large number of bad experiences I've had with them.)
One idea that seems workable is the paid work-sample test. But then you have to do some screening before giving people the test. (I.e., hire someone for a 10-20 hour mini-project before committing to hiring them full-time.)
With 100 hours of free time, I could get a personal project done and put it on the Internet somewhere. That seems like a much better use of my time than doing 10-20 work-sample tests for companies that aren't going to give me an interview even after I do it.
I will second the value of a work sample. It's so simple that it feels incredible that we didn't do them in the past. What my team __hasn't_ is issue the exact same work-sample problem; you make a compelling argument for doing that, so I think that we'll implement it.
While I agreed with many of your points, I could not stop thinking what a huge time burden of implementing something like what you propose, at scale. For a growing company with several work streams and projects hungry for talent, the interview approach you posit would never work.
Another thing that came to mind is the fact that educators and cognitive scientists have been working for decades in what constitutes a good and effective test. Here you seem to claim that somehow you and your folks have "cracked" the system and have come up with a test process that's guaranteed to yield good long-term and performant workers?
Finally, I feel this approach while intended to alleviate some of the most common pain points of the interview process, it tries to reduce it to a number. While it would seem a number is plain and objective, much like a test score, it neglects to "tell the story" and unless all you want are gifted and highly skilled human automatons, you need a way to gauge "soft skills", which one would argue are as important as their core technical competency.
[1] http://mhutter.org/papers/Mulder2014UsingBleichenbachersSolu...
Not sure if there is any difference in content.
Please please please apply for full-time when you graduate! The process is a lot smoother.
Couple questions for you. How do you go about choosing which metrics go into a grade when designing the work sample test? Should the process of grading it be completely automated, so as to eliminate bias? What's the ideal length of time for a work sample test, in your experience?
I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions.
And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".