This is exactly correct. A work-sample test needs to capture the normal parameters of the work. If team members aren't routinely called on to solve programming problems in a high-stakes stick-the-landing-or-you're-fired exercise on a whiteboard in front of an audience, then the prediction the test is trying to make is confounded by all those factors.

* Eliminate live audiences

* Let people work in the environment they'll be able to choose on the job

* Ideally, let people work in their own comfortable environs, even if they won't be able to do that on the job

* If you're worried about cheating, build that assessment into your in-person followup interview

How often did people cheat in remote work-sample tests?

I never once saw it happen.

I helped a friend cheat.

Originally he'd simply asked if I could be there whilst he did the test so that he could talk aloud and bounce ideas off of me as it was timed and he gets nervous in tests and was afraid he'd not think clearly.

He froze. Totally.

I took over and did the test for him, and he aced it and was offered the job at the top salary band.

He is a perfectly good engineer and the company were very satisfied with their hire, but he never did that test. The test in it's entirety was completed by myself.

I cannot imagine this is such a rare thing with remote technical tests.

I believe both of you, but our process did/does nothing to overtly catch cheaters, we hired directly off its conclusions, we hired at a rate faster than most VC-funded cash-flow-positive YC companies, including in SFBA, and we never let anyone go (nor did anyone ever quit while I was there) once we made an offer.

(Matasano is also not a company where it's easy to duck attention and coast; the tempo is 2-3 week engagements that wrap up with metrics that everyone cares deeply about).

The conclusion I draw is that cheating just isn't as big an issue as people think it is.

I've seen it several dozen times. A solution to one of our sample tests made it out to Github and that was all she wrote. For a while we kept the same sample test as a honeypot, but disqualifying half of the candidates for cheating was tiring.

I still agree it's the best approach.

> disqualifying half of the candidates for cheating was tiring.

That actually sounds awesome. You could reduce your interview evaluation overhead by nearly 50%!

Meh. IT unemployment has been about 2% for YEARS. I can get a good job on a team I'll enjoy working with in less than a week. If you're a top tier IT name then it might be worth jumping through hoops, but I will wind up at companies with the least hassle in the hiring department. 15 years in the industry has told me that there's no strong correlation between hiring practices and team quality.

Precisely why I hit the back button if my employment application is through something like Taleo. My time is valuable, too valuable to waste on a resume rabbit hole.

actually, that brings up a really great point. This could be an interesting article. Considering how much the web focuses on 'conversion of web leads' and making the entire process seamless, you'd think you could quantify the candidates lost (and put a dollar figure on it) simply because the interface wasn't friendly and turned away candidates. However, all too often i feel that the goal for everyone involved but the hiring manager, is a seat filled.

Anecdotally, I was prompted 'Zip code required' on a multinational company talent portal. Tried to fix and submit for at least 60 seconds before finally had to modify the CSS to show the input so I could put in a value, as Zip input was set to display:none. I figured at least there would be less competition for the role.

It's possible that your application was the only one... in months... so nobody was even looking at these applications.

You did not get reply in the end, right?

not a peep :)

That's a killer idea for a front-end developer test.

Front-end developer may not want to work in the place that makes such silly mistakes.

If you deliberately made it that way and told your candidates, it might work well. If you're a web dev and can't figure out how to un-hide a field, then you don't deserve an interview.

Definitely intentionally, more of a gauge of problem solving ability. Fix the bugs, submit the test, get interview.

Similarly, I send my resume out as a PDF. If they ask for it in Word format, I know it's going into that kind of system. I may or may not send it in Word format, but it's definitely a mark against them.

I have questions for both you and IndianAstronaut, then. I see statements like this frequently. I also see people talking about desiring to work at places like Intel, Qualcomm, Nvidia, and other such large tech companies that violate one or both of your rules. I assume you are not in that group? Would you really turn down one of those companies of a recruiter approached you and asked for a Word resume or told you to fill out a Taleo profile?

I kind of hinted at that in my original post. If it's a named company that I explicitly desire to work for then I might be willing to jump through a couple of hoops. Generally speaking, though, the name on the sign is not a strong indicator of whether or not I'm interested in working there. I'm usually looking for some combination of salary, technology stack or cool project, and strong/intelligent team. Having a recognizable company name on my resume is a very very low priority.

If a recruiter approached me, then sure. The recruiter is at least keeping an eye on my resume. Otherwise I will not apply. Taleo is a black hole.

Same here! It's a definite alert.

Even more irritating is that recruiters don't appear to know the difference between C#, C, C++ and Obj-C.

> weeds out those that may need a lot of hand holding early on.

i.e. You are using interviews in order to avoid paid in-house training.

True, but theres a difference between paid in-house training and being able to investigate problems on your own without constantly bothering everyone on the team to do your thinking.

The latter is a candidate I try to avoid like the plague.

I'd agree with everything but "alone". While having an interviewer (or more than one) staring at you while you solve a problem is unnerving, I'd also expect to be able to elicit requirements during the exercise, either before, during or near completion - because that's how it has always happened on my past assignments.

Do you let your interviewees reach out (possibly via IM) to your interviewers with specific questions?

Yes, we all have skype on our machines and I show them how to dial my phone extension if they need anything.

Edit: also for clarification, we discuss the problem ahead of time to make sure they understand it and ask any questions. For my group it's always a basic full stack problem. Here's a mockup and a basic install of visual studio and MSSQL, show me you can CRUD some data and present a decent UI. then we'll talk about how you did it.