Re:The Utter Uselessness of Job Interviews
by peripitea on Sunday April 09, 2017 @04:32PM
(#14074224)
Making sure I understand you: When you refer to work sample tests here, that refers to the crypto challenges and things like that that you published at Matasano? And you're saying that was much more predictive than the list of facts methods, right?
That's what people ordinarily assume I mean, but while our work sample challenges were similar to the cryptopals and Microcorruption stuff, they were not the same, or even derived from them. They were designed specifically to qualify candidates, and in fact predated our public challenges.
Re:The Utter Uselessness of Job Interviews
by sgustard on Sunday April 09, 2017 @08:15PM
(#14075065)
Just to further clarify, by work sample you do NOT mean an example of previously produced work? This still seems fickle: my dozen years of work output (for example, building and running a site with perfect uptime for millions of users) is not as valid an indicator of my future performance as how I happen to score on some arbitrary timed test?
Absolutely not. Samples of previous work are deceptive at the best of times.
Re:The Utter Uselessness of Job Interviews
by peripitea on Sunday April 09, 2017 @08:01PM
(#14075018)
Ahh, interesting. Would you mind sharing an example of how they were different, or how the fact that they were specifically for qualification changed the thought process? I'm working on making tech interviewing better and am fascinated by this stuff.
NDAs? Everything I've done yet is proprietary and judiciously covered by various binding legal agreements.
I wouldn't breach such a contract nor should you hire someone that would be willing to do so. The sole exception being in the cases where it would be the best for the public interest, i.e. whistle-blowing.
The reality is that generating good questions for a structured interview is difficult. You can't just pose a programming problem. As you've noted, most programming problems have multiple good answers. Differentiating between multiple good answers from different candidates re-introduces subjectivity. Your brain would rather convince you that one valid answer is less good than another, even when it's not, that to admit to you that it can't differentiate or generate a narrative for you. The part of your brain that generates narratives is incredibly powerful and does not care about how accurate your hiring process ends up being.
What we tried to do was create questions that generated lists of facts. "Spot all the errors in this code" would be an example of this approach (but none of the three we used). We went into the process wanting to embrace epistemological uncertainty, generating a historical trail of data that we could retrofit to candidate performance.
In the end, work sample testing was so much more powerful a predictor for ourselves that we never fully got around to analyzing the data. Sometimes we'd get candidates that clearly generated inferior "lists of facts"; I think there may have been 1-2 instances where that outcome actually overruled work-sample testing delivered prior (out of a few tens of engineering hires and probably ~100 interviews).