If you had asked at Matasano how we evaluated our work-sample tests, we'd have told you. In fact: the first contact any new candidate had at Matasano was a 45-minute-long phone call with a director+ (for a year and a half: me), during which we explained our process in excruciating detail and answered any questions that came up. If we got even a hint from a candidate that they might not know exactly what we were looking for, we'd get their mailing address and fedex them a stack of free books; we gave them cheat-sheets on what parts of those books to read, as well.

I have the strong impression that a typical SFBA tech hiring manager thinks it's crazy to give candidates cheat sheets for interviews, and that helping candidates with technical evaluations decreases rigor. THE EXACT OPPOSITE THING IS TRUE. Evaluating technical ability under unrealistic pressure and with unrealistic restrictions confounds the results of the evaluation.

Of course, you have to design tests that are valid predictors for candidates who have received tips and assistance and practice and resources. That sounds hard! But it isn't. The best predictor for a candidate's ability to do the kind of work you really do is, simply, the kind of work you really do. Not problems from a book, not trivia questions, not one algorithm poorly implemented on a shared coding site and a Skype call, not whatever bug is at the top of the bug tracker today, but a sample assignment given to a real member of your team, the same one, for every candidate, graded the same way.