News for hackers. Stuff that matters.
Re:The Hiring Post by tptacek on Friday March 06, 2015 @05:01PM (#9159878)
Author.

I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions.

And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".

Re:The Hiring Post by Micaiah_Chang on Friday March 06, 2015 @07:23PM (#9160405)
Have you ever read The Checklist Manifesto[0]? I may be reading too much into this post, but the lessons you learned from this interview process have frighteningly close parallels to the lessons in the books. I doubt the book had any influence on your interview process, seeing as it was published after the interviews were formalized, but the book seems like it might have new lessons.

For example, a good portion of doctors absolutely hated using checklists. Yet, when pressed, readily admitted that it prevents simple mistakes and that they would prefer to have them rather than not to. Another is that entries that address more human concerns, e.g. "Have everyone introduce themselves", have a place on good checklists.

[0] http://www.amazon.com/Checklist-Manifesto-How-Things-Right/d...

Re:The Hiring Post by calinet6 on Saturday March 07, 2015 @11:32AM (#9162192)
Extremely true. This is about systematizing the right things about interviewing, and making them solid. Checklist Manifesto is all about that, a great simple way to create systems that work.

At a high level, this is all about Deming ( http://en.wikipedia.org/wiki/W._Edwards_Deming ) and TQM concepts -- if you want to achieve a high-quality output, measure the things that matter, and understand the variation present in the system. Once you have a stable system with good data achieved by good methods, you may then begin improving it. Attempting to improve a complex system without knowledge results in unpredictable changes—we call that tampering. Simple but beautiful.

So, in essence, this is an extremely natural and correct application of quality management principles to the hiring process. Stellar.

Re:The Hiring Post by platz on Saturday March 07, 2015 @12:17AM (#9161083)
Re:The Hiring Post by cyounkins on Friday March 06, 2015 @05:41PM (#9160077)
How do you avoid wasting time on candidates that are a poor fit? When we advertise for an open position we get a ton of unqualified candidates.

In your post you say, "At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began." You seem to continue on from that point. What happens before first-call?

Re:The Hiring Post by tptacek on Friday March 06, 2015 @05:44PM (#9160083)
To me, recruiting is two problems: outreach and qualification. You're asking about outreach. How we did outreach is a whole 'nother blog post, and one I'm bound to write soon.

What surprised me about recruiting was how much more important qualification is than outreach. Without good qualification, it almost doesn't matter how good your outreach is, because you're filtering for the same highly visible easily accessible candidates every other firm is looking for. The best-priced talent is buried. What you need is ground-penetrating radar. Clever outreach schemes helped us, but what really let us make our best hires was a qualification process that allowed us to confidently ignore resumes.

We didn't just maintain our team's level of quality by ignoring resumes. We transcended it. We never could have run a resume-based process that would hire people who could use lattice math to break crypto. To hire those people, we needed to select for aptitude, not experience.

I haven't said it anywhere yet, but in case the subtext isn't clear: the person who can implement an attack for which edit/compile/debug takes 4-6 hours also does a pretty great job on every other facet of the software security job. There are aptitudes that are very good proxies for other aptitudes!

Re:The Hiring Post by cyounkins on Friday March 06, 2015 @06:00PM (#9160156)
Maybe I wasn't clear with my question. Say the company indicates it is hiring and gets 100 interested people. How do you filter them down to serious applicants without looking at resumes? I would argue that isn't so much outreach as it is qualification, as the people are already interested.

Maybe my premise of 100 candidates is incorrect based on how you did outreach?

Re:The Hiring Post by jerematasno on Tuesday March 10, 2015 @06:45PM (#9181279)
At Matasano (slash NCC), we get a lot of candidates. We look at resumes so that we have something to break the ice with when we talk to the candidates. We do triage interns using resumes, because we get hundreds over the course of a few weeks, but never for full-time candidates.

How do we figure out which ones are serious? We talk to them, usually give them homework, and then give them work-sample tests. This is not particularly expensive for us, and gives us repeatable, relevant, apples-to-apples metrics on how qualified they are to do the work. This is, as Tom insists, surprisingly easy for us. (Not to say it's easy, but nothing about hiring is easy.)

Re:The Hiring Post by AceJohnny2 on Friday March 06, 2015 @06:28PM (#9160244)
tptacek mentioned in some other comments that Matasano runs ongoing "Capture-the-flag" contests/games (such as [1]) that basically self-selects interested and interesting candidates for them.

[1] https://microcorruption.com/login

Re:The Hiring Post by mrkurt on Saturday March 07, 2015 @12:49AM (#9161144)
Serious is "testable", in the sense that you can ask them a specific question about the job to see if you get a response. We warm people up on the phone when they're putting special attention into our positions, and give everyone the opportunity to answer work samples (whether they seem serious or not). The point of warming them up is to balance out the effort — we're asking people to spend a few hours on this stuff, it's fair to give them time.
Re:The Hiring Post by joepie91_ on Friday March 06, 2015 @10:49PM (#9160913)
Do you really need to "filter down" your candidate list, though? Given the apparent importance of hiring processes, it seems to be like it may well be worth it to simply go through the process with all 100 of them.
Re:The Hiring Post by tempestn on Saturday March 07, 2015 @02:42AM (#9161328)
One possibility is to make a very simple but non-standard request in the posting - for example, to submit the answer to an extremely straightforward coding question or some such in their cover letter. That would filter out a lot of the 'shotgun approach' applicants immediately.
Re:The Hiring Post by rdtsc on Friday March 06, 2015 @05:23PM (#9159989)
Very good post. Thank you for sharing it. I've been interviewing applicants quite a bit lately, so have been thinking about these issues.

Two questions for you.

What is your experience hiring fresh college grads? Is the process different? Since one of goals of the process seems to not arbitrarily adjust it for every candidate, but rather stay consistent, do you think it makes sense to adjust it for someone who just graduated. They might have been learning computer science type stuff, data structures, did a few group projects and have certainly been practicing the typical interview puzzles, fizbuzz and O(n) times of the all data structures.

How do you account (or do you account at all) for culture fit. That is a fuzzy and slightly dangerous area. Is there a score in that regard? There are some people who are brilliant but they do not work well in a team, or rather would not well in one particular team. This is course full of danger as "culture" fit often becomes an opaque discriminator for racial, sexual and other stereotypes. "He's just not a good culture fit" is such an overload euphemisms that I feel kind of sleazy just saying.

Re:The Hiring Post by tptacek on Friday March 06, 2015 @05:30PM (#9160031)
Everyone has exactly the same process, with the exception of interns. Interns have an abbreviated interview (shorthand: we switch from assessing aptitude to assessing enthusiasm), and any time one came back looking for a full time job, they got fast-tracked. I'm not saying that was the best policy (it did keep us from collecting some extra data), but by the end of an internship you had a really good idea that you wanted to work with someone.

The lede I buried in this post is that we hired more-or-less resume-blind. I had final call on every hire and went through some effort not to look at resumes. On our first-calls, I'd ask some questions that would give me background hints about candidates, but that was primarily to tune my spiel (I didn't want to explain blind SQL injection to someone who'd spent 3 years as a pentester already, for instance).

If I had it to do over again at Matasano, I'd have done it formally resume-blind. Only Dina would have had access to resumes, and interviewers would be forbidden to ask resume questions. The process would have worked better if resumes were firewalled out completely.

There was no culture fit score; in fact, if you gave subjective feedback ("passion", "confidence", "work ethic") your feedback was likely to be discounted. We were proud of the culture diversity we managed to scrounge out of our candidate pool; to have team members with kids who needed to keep strict hours, and team members who'd roll in at 11 and still be there at 8 working on a pet problem. We had drinkers and non drinkers, college students and people with 15+ years of dev experience.

You could get booted from the process for being overtly an asshole (I, for instance, might not get hired), but that happened very rarely and with crystalline clarity.

Re:The Hiring Post by enraged_camel on Friday March 06, 2015 @08:44PM (#9160619)
It's interesting that you make an effort to not read resumes. Do you explain that to people you interview during the first call? The reason I ask is that, if a hiring manager asked me things that are already written on my resume (e.g. what did you study in college?), I would definitely be annoyed, as my default expectation is that they should read the resume before the interview.

Don't get me wrong, I understand and agree with your reasoning. I'm just curious how transparent it is to the other side.

Re:The Hiring Post by tptacek on Friday March 06, 2015 @08:59PM (#9160653)
We weren't super transparent about it, but we tried really hard in a lot of other ways to take hiring seriously as a customer service problem, so maybe we came out ahead anyways.
Re:The Hiring Post by enraged_camel on Saturday March 07, 2015 @12:15AM (#9161077)
Thank you for the response Thomas. That makes a lot of sense.
Re:The Hiring Post by kasey_junk on Friday March 06, 2015 @09:24PM (#9160708)
I've been arguing for a resume blind hiring practice for years and one of the kickers is that you also need a resume question blind approach.

So really "what did you study in college?" needs to come off the list of things you ask (at least as part of the filter).

Re:The Hiring Post by jerematasno on Tuesday March 10, 2015 @06:51PM (#9181323)
The way we do it now, the initial call person reads your resume, but we categorically do not reject based on the initial call (or the subsequent tech phone interview(s)). So, we get the advantage of being able to talk to someone about their experience and helping them make a plan to get hired, but also the advantage that our process is actually resume-blind.
Re:The Hiring Post by rhizome on Friday March 06, 2015 @11:26PM (#9160983)
That's a red flag on the company though, isn't it? Why would they be asking you to recount what's on your resume?
Re:The Hiring Post by spydum on Saturday March 07, 2015 @09:27AM (#9161861)
this is true and at many companies the hiring process is so broken, the interviewer might have only an hour or less heads up that they will be interviewing someone. i have done that maybe twice, and since then i refuse tomdo interviews on candidates without more prep time.

on the reverse of that, i have sometimes avoided topics that the candidate was familiar with because i didnt see relevant experience on a resume. that was a major miss, and ive tried to avoid using resumes for shaping the interview topics since.. i see the resume now as more of a signal of what the person is interested in (since most of us carve huge swaths of our experiences out, and tailor them for the job we apply to anyways).

Re:The Hiring Post by rhizome on Saturday March 07, 2015 @01:24PM (#9162549)
When I was at EA, I would get an email saying I had to interview someone that afternoon, here's their resume, and here's the title they're interviewing for. It was ridiculous.
Re:The Hiring Post by rdtsc on Friday March 06, 2015 @06:12PM (#9160200)
Thank you for taking the time to answer my questions.
Re:The Hiring Post by tptacek on Friday March 06, 2015 @06:13PM (#9160203)
Thank you for asking interesting questions!
Re:The Hiring Post by tomp on Saturday March 07, 2015 @07:33AM (#9161695)
Such asshole. I can totally see why you wouldn't get hired at Matasano. /s
Re:The Hiring Post by jazzdev on Saturday March 07, 2015 @10:28PM (#9164383)
How do you access enthusiasm?
Re:The Hiring Post by nissimk on Friday March 06, 2015 @05:17PM (#9159951)
It's funny, because to change it, you have to admit that the way you were doing it before is not effective. I've come across many people who admit that they are bad at being interviewed, but very few people I meet have admitted to being bad at interviewing others. It's incongruous with the results, because these are the same people that have hired folks that we all agree are not exhibiting the level and characteristics that they are supposedly screening for.

Good job coming up with something different. I hope it's really working out as well as you say.

Re:The Hiring Post by tptacek on Friday March 06, 2015 @05:22PM (#9159979)
I was horrible at interviews. I was a sport interviewer. I had favorite tricky questions. "Design the fastest possible traceroute" was one for a long time. "Describe the most important feature of TCP NO WRONG, IT'S CONGESTION CONTROL NITWIT". I trust my own interviewing ability least of all, which is why I sink so much time into avoiding them.
Re:The Hiring Post by pcthrowaway on Friday March 06, 2015 @06:28PM (#9160246)
Why do you feel that congestion control is so much more important than guaranteed delivery? Guaranteed delivery would have been my first thought.
Re:The Hiring Post by tptacek on Friday March 06, 2015 @06:32PM (#9160255)
Guaranteed delivery is extremely easy to provide. TFTP manages it. But think about a bunch of ethernet-connected browsers sharing a 1mb/s link; that's kind of a miracle.
Re:The Hiring Post by throwawaykf05 on Sunday March 08, 2015 @10:30AM (#9165674)
> "Design the fastest possible traceroute"

Hmm, I once took it upon myself to do exactly that (mostly using non-blocking sockets, skipping hostname lookups, etc.), but what's so "tricky" about this question? Assuming the position involved network programming, I'd consider it a good way to test someone's networking knowledge. Is the "tricky" part knowing how routers often drop ICMP replies if they exceed some threshold rate?

Re:The Hiring Post by powertower on Saturday March 07, 2015 @03:43PM (#9163050)
> I was horrible at interviews. I was a sport interviewer...

That's nowhere near as bad as the interviewer who has the primary goal of demonstrating how smart he is (to the person he is interviewing), with the actual hiring process being secondary.

Re:The Hiring Post by tptacek on Saturday March 07, 2015 @03:58PM (#9163113)
That is automatically what I think about anyone who says "I'm a very good interviewer".
Re:The Hiring Post by jfb on Friday March 06, 2015 @09:21PM (#9160706)
I am painfully aware of my limitations as an interviewer. I wish there was institutional support for interviewing as a skill distinct from more specific job requirements. I've worked at big companies, small companies, been a founder, and I've never been comfortable with engineering interviews. So, more power to tptacek in his crusade.
Re:The Hiring Post by bentcorner on Friday March 06, 2015 @05:35PM (#9160059)
I'll admit that I'm bad at interviewing others. I'm not trying to humble-brag or anything. I'm just bad at it. Part of it is that I haven't had to go through an interview in over 10 years, so it's difficult to empathize with a candidate. I do my best, though. The other part is that I'm happy at what I do, but I'm not the best. I sometimes can't keep up with candidates when they're writing on a whiteboard and make a claim that such-and-such works. I squint and can make a judgement on that claim, but really I need to cover x, y, and z during our hour together and we're already 20 minutes into it and I don't want to get dragged into the weeds.

So, I'm not a fan of giving interviews. I wouldn't mind shadowing someone better than I, but my first inclination is that that would be a terrible idea since it would make candidates even more nervous.

Re:The Hiring Post by aristus on Friday March 06, 2015 @05:13PM (#9159935)
This is a good post, Thomas. From this I think I could make an efficient test for the kind of person who'd make a good performance engineer. Developing a more general work-sample for developers is a harder problem, but still feels doable.

This may not be an issue for you yet, but what about plagiarism? And can you expand on your hints about how you changed your pipeline at the front of the process? You don't interview every resume that comes your way, right?

Re:The Hiring Post by tptacek on Friday March 06, 2015 @05:18PM (#9159959)
Every serious application, regardless of their resume, got a 30-45 minute phone call. You post hiring posts all over the place and you get a lot of unserious submissions, but an easy way to screen them out is to respond with a question, like, "Thanks for writing! We'd be happy to talk to you. Can we ask what has you interested in doing software security?" The answer doesn't matter, as long as there is an answer.

I did a lot of calls where I knew a couple minutes in that we were unlikely to hire the person, but (a) I got surprised by outcomes enough not to shirk on those calls, and (b) those calls are a very small price to pay for finding buried talent.

Re:The Hiring Post by kasey_junk on Friday March 06, 2015 @09:30PM (#9160721)
I also advocate for a work sample based hiring pipeline and the plagiarism question often comes up. My response is always:

1) People generally don't cheat. If you have a high percentage of cheaters its how you are filling your pipeline you need to address, not the filter.

2) What most people think of as "plagiarism" is actually very common in the real work of software developers. Very frequently you see/mimic other peoples work to solve problems. Instead of being freaked out about a skill that is central to the job, why not use it to evaluate the candidate. Did they "plagiarize" the right thing? Did they do it effectively. Did they do something backwards that a simple google search would have found a thing to copy?

3) If you are big enough for plagiarism to be a real problem and have addressed points 1 & 2, it is relatively easy to detect mechanically (and there is a surprising amount of research in the field as CS professors invariably write both an automated grader and then an automated cheater detector).

Re:The Hiring Post by tomp on Saturday March 07, 2015 @08:04AM (#9161734)
Hi Thomas. Great post!

I was wondering if you could share some info on the salaries. Did you offer every candidate the same salary or adjust it (based on resume/previous salary/age/experience/...)? You mentioned 100% retention - so I assume no candidate ever left for another position with a higher salary?

Being a specialised consulting firm, I imagine Matasano could offer highly diverse, very interesting work long-term (because consulting), and higher-than-average salaries (because specialised), so maybe it wasn't a problem for you to offer higher salaries than the competition, or to retain candidates at equivalent salaries.

Re:The Hiring Post by knite on Friday March 06, 2015 @05:28PM (#9160014)
Thomas,

Great article. The prelude to the interview with books given to candidates is especially interesting. I'd love to see that reading list, and would further like to suggest posting it on your careers page. I would imagine that anyone who goes to the trouble of working through your suggested materials would be providing a very strong signal of quality and investment.

Re:The Hiring Post by tptacek on Friday March 06, 2015 @05:30PM (#9160034)
Here's a version of it: http://amzn.to/cthr46
Re:The Hiring Post by zodiac on Friday March 06, 2015 @10:44PM (#9160899)
You mention that every serious applicant gets a 30-minute "first call" - what percentage of applicants get this call? How do you decide which candidates get this call?

Also, what percentage of people who complete the work-sample task get an onsite interview?

I'm asking these questions because I think most software companies who filter out people based on resumes and phone screens do it mainly because there are too many applicants and they need to filter out many candidates at earlier stages. Do you think your hiring strategy resulted in more time spent on the process per successful candidate and per applicant?

Re:The Hiring Post by fsk on Friday March 06, 2015 @11:53PM (#9161042)
As a candidate, I've soured on work-sample tests. Too many times, I've done the assignment, and not even gotten an interview (and I know I submitted a correct answer).

As someone who already has a job, committing a large number of hours for the chance of an interview seems like a waste of time.

If every employer demands a 5-10 hour work sample test before they even talk to you, that really isn't a scalable solution from the viewpoint of a candidate. I could easily put in 100 hours into works-sample tests without getting a single decent interview, so now I just refuse them.

Re:The Hiring Post by Offler on Saturday March 07, 2015 @04:20AM (#9161427)
They can be enjoyable tests. I have learnt new things while doing some.
Re:The Hiring Post by log_n on Saturday March 07, 2015 @11:53AM (#9162277)
Yeah, those are the ideal. A lot of us are already learning different things during our down-time. An enterprising company could take that drive to learn new things to find potentially good employees by 'tricking' them into learning something new from them.
Re:The Hiring Post by tptacek on Saturday March 07, 2015 @10:23AM (#9161964)
Just a reminder: work-sample testing isn't the entirety of my recommendation, or even the majority. Employers need to do more than one thing to fix the hiring process.
Re:The Hiring Post by fsk on Saturday March 07, 2015 @04:47PM (#9163241)
I'm beginning to think that one way to "fix" hiring is to do stuff that nobody else is doing.

Everyone else is screening people based on resumes? Then, you can find good candidates by ignoring resumes, so you find good people with bad experience.

Nobody else hires people older than 40? Then you target people who are older than 40 but are still good workers.

Everyone wants a programmer with X experience but there's a shortage of people with X experience? Then hire good programmers without X experience, and give them a chance to learn X on the job.

Nobody else gives a work-sample test? Then giving a work-sample test is a benefit. You find candidates who are desperate and good enough to do the work-sample test. If everyone gives a work-sample test, then you get an advantage by NOT doing a work-sample test. (For example, a company with a work-sample test is now excluding me from their candidate pool, due the the large number of bad experiences I've had with them.)

One idea that seems workable is the paid work-sample test. But then you have to do some screening before giving people the test. (I.e., hire someone for a 10-20 hour mini-project before committing to hiring them full-time.)

With 100 hours of free time, I could get a personal project done and put it on the Internet somewhere. That seems like a much better use of my time than doing 10-20 work-sample tests for companies that aren't going to give me an interview even after I do it.

Re:The Hiring Post by wtbob on Friday March 06, 2015 @05:55PM (#9160128)
Just wanted to let you know that I really enjoyed reading your article and will probably crib shamelessly from it in the future☺

I will second the value of a work sample. It's so simple that it feels incredible that we didn't do them in the past. What my team __hasn't_ is issue the exact same work-sample problem; you make a compelling argument for doing that, so I think that we'll implement it.

Re:The Hiring Post by thalesfc on Saturday March 07, 2015 @12:58AM (#9161166)
Amazing text, last month I have screwed up at an interview just because I had no knowledge about what they were asking. After the interview I google about it for 2 minutes and then I was able to quickly answer that question.
Re:The Hiring Post by aminox on Saturday March 07, 2015 @06:19AM (#9161596)
Let me first start by saying that I, too, dislike the prevailing interview process for software development jobs.

While I agreed with many of your points, I could not stop thinking what a huge time burden of implementing something like what you propose, at scale. For a growing company with several work streams and projects hungry for talent, the interview approach you posit would never work.

Another thing that came to mind is the fact that educators and cognitive scientists have been working for decades in what constitutes a good and effective test. Here you seem to claim that somehow you and your folks have "cracked" the system and have come up with a test process that's guaranteed to yield good long-term and performant workers?

Finally, I feel this approach while intended to alleviate some of the most common pain points of the interview process, it tries to reduce it to a number. While it would seem a number is plain and objective, much like a test score, it neglects to "tell the story" and unless all you want are gifted and highly skilled human automatons, you need a way to gauge "soft skills", which one would argue are as important as their core technical competency.

Re:The Hiring Post by tptacek on Saturday March 07, 2015 @10:22AM (#9161961)
Actually, go look for the 'tokenadult comments on work-sample tests to learn that this approach has been known for something like half a century to outperform interviews.
Re:The Hiring Post by aminox on Saturday March 07, 2015 @01:01PM (#9162489)
This answer could possible address the test question. It leaves the other two issues I raised unanswered. Not that you have to address them, but just saying that the approach you guys implemented might work well for a very small subset of companies that share traits in terms of company size, structure and industry sector.
Re:The Hiring Post by tome on Sunday March 08, 2015 @04:07PM (#9167003)
Re:The Hiring Post by gone35 on Saturday March 07, 2015 @08:40AM (#9161788)
So what was the paper? De Mulder et al. 2013[1]? (Sorry not my field; just a guess!)

[1] http://mhutter.org/papers/Mulder2014UsingBleichenbachersSolu...

Re:The Hiring Post by sdevlin on Saturday March 07, 2015 @09:19AM (#9161849)
Yep. I remember seeing this version though: http://eprint.iacr.org/2013/346.pdf.

Not sure if there is any difference in content.

Re:The Hiring Post by gone35 on Saturday March 07, 2015 @09:32AM (#9161867)
Thanks! The one I linked to is just the extended version.
Re:The Hiring Post by chloridite on Friday March 06, 2015 @06:21PM (#9160224)
How often is it that a candidate shows positive on work-sample tests but then shows negative after a face-to-face interview? Should you explain negative outcomes to the rest of the company or to the candidate?
Re:The Hiring Post by tptacek on Friday March 06, 2015 @06:35PM (#9160267)
I can think of a couple times when work-sample overrode face-to-face. I'm sure face-to-face overrode work-sample at least once (we had a scoring system, and the threshold for getting a face-to-face interview was not "perfect score"; you could be "on the bubble" and get an interview), but to my knowledge it never overrode a very good work-sample score.
Re:The Hiring Post by thewarrior on Saturday March 07, 2015 @04:21AM (#9161430)
Thank you for this article. I found it motivational in a way. Especially when many of us think there is not much more room to grow and we feel terrified of whiteboard interviews at the bigger cos.
Re:The Hiring Post by lilpirate on Saturday March 07, 2015 @02:09AM (#9161284)
I applied for an internship at Matasano. I was asked about office preference and VISA sponsorship. Now I feel bad I didn't get an interview.
Re:The Hiring Post by jerematasno on Tuesday March 10, 2015 @06:57PM (#9181357)
Getting an internship at Matasano is HARD. Unlike normal hiring, we are limited in the number of spots we can offer, and we also have a huge flood of candidates at once. I hate that it's so hard, and that we can't provide our usual standard of support to interview applicants. There's just no way for us to scale with the seasonal demands for summer interns.

Please please please apply for full-time when you graduate! The process is a lot smoother.

Re:The Hiring Post by Kaedon on Saturday March 07, 2015 @12:40PM (#9162419)
Very interesting, thanks for sharing! It made me think about how we could improve our process.

Couple questions for you. How do you go about choosing which metrics go into a grade when designing the work sample test? Should the process of grading it be completely automated, so as to eliminate bias? What's the ideal length of time for a work sample test, in your experience?