I'm still not sure how giving employees a book to read and then testing them on it is a good process. If I'm going to spend a couple months of free time learning something, I'd pick something like Python or Android development or HTML5 app development, rather than learning a skill that will only be useful for 1-3 employers.

How can I know if an employer is worth 1-3 months of my free time until I meet my potential future coworkers? Demanding a candidate invest 1-3 months of free time before you meet them seems like an insult to me.

If your job uses a niche skill, why demand candidates be an expert already before you hire them?

I did not have a problem with passing on candidates who weren't not sold on learning our subspecialty.

I do have a problem with passing on candidates who were sold on our subspecialty, had an aptitude for it, but could not pass an interview on it "cold".

To me, that sounds like a convincing argument that your specialty is not worth learning. I'd rather work someplace that wants to invest in their employees, rather than expecting them to already be experts in some obscure techniques.

Are you hiring people who are brilliant, or people who are so desperate that they'll spend a couple months preparing for one interview?

Let's not flee to abstraction. Matasano/NCC does software security. We were willing to invest some time and money to bring people up to speed in software security and exploit development.

Not your cup of tea? Totally fine. Not everyone is interested in doing security.

I like this summary and plan on stealing it from you.

@tptacek, I'd be interested in your thoughts on how one could adapt your methods from Matasano to, say, a startup doing web dev or to an enterprise software company. I've done some thinking based on some of your writings and the idea that the best way to test someone for suitability to do a job is to have them do the job, and haven't really come up with any good ideas.

1. Take application you've already built.

2. Package it, with all of the assets and utilities needed to get it running with "vagrant up".

3. Carve out some feature/features from the application, and replace them with stub functionality.

4. Deliver the vagrant app and a functional spec to candidates. Have them implement the missing feature.

5. Devise a scoring rubric (unit test coverage, lines of code, algorithms used, safe/unsafe APIs, performance, whatever). Mechanically evaluate candidate submissions.

6. (Optional) Devise a 15-20 minute on-site interview component to verify that the candidate actually did the work. We didn't bother with this, and multiplied the size of our team (NCC is the largest software security firm in North America) and had 100% retention. But it's a big concern for some people.

I remember back when I was a teenager looking at colleges seeing a private tech college with a high acceptance rate and a public state college with a low acceptance rate. The Princeton Review had rated the private tech college as much more selective than the public state college. Teenage me was confused - how was that possible? - the state school had a significantly lower acceptance rate!

The reason is, of course, self-selection bias. On average, the people who applied for the tech college were just stronger students than the people who applied to the state college. This likely is because applicants to the tech college were more likely to be people whose primary goal was to peruse an academic discipline that interested them while the state school had a good number of students applying who wanted to party and attend football games. If you were to look at students who applied to BOTH schools, you would actually see a lower acceptance rate for the tech school than the state school.

I suspect that the same factor is at play here. I suspect that the average quality of a candidate who applies for a software security engineering position is much higher than the average quality of a candidate that applies for a enterprise software development position. The software security engineering position is an esoteric position that is more likely to attract applicants who are enthusiasts or at least very interested in the field. The enterprise software development position is more likely to attract anyone with a tech background looking to phone it in and collect a paycheck.

If this is true, while both Thomas and Steve have a difficult time hiring developers, they have a difficult time for fundamentally different reasons. Thomas is like the tech school - his difficulty is with getting candidates to apply. Steve is like the state school - his problem is with separating qualified candidates from "phone it in and collect a paycheck" candidates.

This is why I am skeptical of Thomas thinking he's got a much better way of hiring figured out. Maybe he does for companies that focus on security - judging by his Linkedin profile, he's only worked for companies focused on security. This is certainly valuable, but he has no idea what it is like for Steve.

By the way, I think many of us have seen this self-selection bias at play even within our own companies. Post a primarily Java job and a primarily Scala job and then try to tell me that the same principles apply for filling both positions.