I think this is why HireArt (YC W12), exists. They pre-vet these candidates using a work sample for you, presumably they only administered and graded the work samples of those they found passed the 'resume bar.'
As for false positives, from talking to many employers the fear false positives is generally low because they figure they can "weed them out" during the interview stage. And while there may be statistical evidence showing a high false positive rate, it is surprisingly hard to convince hiring managers of this.
Everyone who applies is sent a link. That link contains a problem description, and a stub program. You're supposed to write the program and send it back. When you send it back, they compile your program and run a standard set of unit tests on it. If it passes the unit tests, then a human looks at it. Generally the code will be "good enough" to bring the person in for an interview.
In the interview you will ask the person enough questions about their solution to give you confidence that the person you are interviewing actually wrote the code you are reviewing. This greatly shortens the interview process.
This is not a hypothetical approach. I know of more than one company that has actually implemented this.
You might think that it takes a lot of developer time to implement, but it doesn't. What you do is have a developer come up with a reasonable project that they think can be done in a reasonable amount of time. That developer then solves the problem. You take their solution and stub it out - that gives you the initial stubs people can download. The unit tests that they developed are the unit tests that you will use. And now you're off and running.
* Problems that are so interesting that people would hack on them just for the fun of it, even if they have no interest in the job.
* A company that's high profile enough (and in a good way) that they can have slightly abusive hiring practices and still get an abundance of good candidates.
* You're looking for inexperienced or desperate people.
The canonical example of using programming problems as a pre-filter was ITA software, who had both of the first points covered. There's some great discussion on this in the HN archives, e.g. http://news.ycombinator.com/item?id=3429231
That may be the case if you're trying to headhunt experienced developers who are currently in good jobs.
But if you're an unemployed programmer, or an employed programmer actively looking for a different job, why wouldn't you take a programming test? Programming tests and technical interviews where you have to code are normal and expected for coding jobs, at least in my experience.
The hiring process is very expensive. But it's expensive in a fairly symmetric way. It's in the interest of both parties to stop the process if there isn't mutual interest.
Both the company and the candidate will spend a lot of time in interviews, so it's in everyone's best interests to make sure that time isn't wasted if there no (or almost no) chance of the person being hired / accepting a job.
There might be earlier steps in the process, but those are also symmetric. It'll take a few minutes of your time to send a resume to a company, but somebody there will be tossing away 80% of them after reading through them once. There might be a phone screen or two in advance, to decide whether there's any point in having someone come in for an interview, but that is likewise costly to both parties.
But a programming test as the very first step of an interview is totally different. Now the costs are asymmetric. The company has an up-front cost (potentially substantial) but spends little time on each candidate. The candidate spends the time up-front, and only later finds out whether the company is at all interested.
The candidate isn't invested in the process yet. So why would he spend an hour, when he has no idea of whether the company is interested? Maybe it's a really fun problem to solve. Or maybe he really, really wants to work at that particular company. Or maybe he is indeed unemployed like you suggested.
But someone currently employed is likely to have a different priorities from a student or someone who is unemployed, even if they are actively looking for a new job. How large a percentage of them are you willing to weed out completely just by making it harder for them to even enter the hiring funnel? 90%? 50%?
I'd rather be filtered by a programming test, where my programming abilities put me ahead of the pack!
With that said, most of the programming tests I've been given in the past have been fairly interesting so perhaps that biases my opinion. I'm certainly no fan of jobs where they send you a huge MS Word application form.
Programming tests can indeed be interesting. Anyone feeling bored could do worse than to attempt solving something from ITA's hiring puzzle archives. But unless you're extremely certain that your tests hit the sweet spot, it seems really dangerous to make them a mandatory first filter. A test once both parties are at least a little bit invested in the process, sure. Or if resume screening really is a problem, allow someone to "skip the queue" by sending a test solution along with the application.
One of my top concerns if I'm interviewing for a company is, "Are these people going to be sufficiently competent to be fun to work with?" If the interview doesn't provide a good enough filter for incompetence, then I am going to not want to interview there.
If my first experience demonstrates that only competent people will get through, then that is going to improve the company in my eyes.
That said, it will slow hiring down. If you want to hire a bunch of people, you can't put a restrictive filter like that up. But if you just need a few, it can work out in your favor.
if not resumeText.lowercase().find("postgresql"):
reject()
If there is an automated code test, I know exactly how the filtering is going to work. I write code, it passes unit tests, no auto-reject.
Sounds like a win for me. Of course, unlike many applicants, I can actually write code that works.
And while HN is a very programmer centric culture, most jobs can't be automated in this way. Look at the OP's job description, how do you automate that? And how does someone like the OP automate that process? Believe me, I would love to live in a world where every position in every company is able to command a great developer to build a customer filtering solution, but most companies in the valley (much less out of it) won't invest this time/infrastructure. Hence the rise of BS keyword filtering systems. I'm personally very interested in solutions that can make this process better, but it's a highly non-trivial problem in most subject areas.
The recession ended years ago. May 2009, if I remember right.
"Hiring for this position will be based in large part on a work-sample test during a half day at our office"
versus
"Previous experience . . . preferred, but will train the right candidate."
We can tell that the United States economy is in recession (as is the case in many other countries with Hacker News participants) because we keep seeing new stories submitted every day or so about the hiring procedures of companies, with multiple comments. In a long FAQ post I've posted recently
http://news.ycombinator.com/item?id=4270768
here on Hacker News, based on many helpful comments from other participants, I summarize a LOT of research on company hiring procedures. If you want to hire someone good in the United States, make a work-sample test part of your hiring procedure. Work-sample tests are much better than biographical reviews of resumes for finding good workers. If you want to get a good job in a well managed company, develop the skills to get past a realistic work-sample test for the position you seek. Many more details appear in the FAQ,
http://news.ycombinator.com/item?id=4270768
which is quite long but well worth a read if you are looking for a job or if you are a business manager trying to hire someone who will do a good job.
P.S. I still hear of young people who are gaining full-time, full-benefits jobs in today's economy. In the usual case, they are getting those jobs by showing what they can actually do as part of the hiring process, degree or no degree.
After edit: I recently read the VERY interesting book The Folly of Fools: The Logic of Deceit and Self-Deception in Human Life
http://www.amazon.com/Folly-Fools-Logic-Deceit-Self-Deceptio...
(I wonder if this book has been discussed here on HN yet?) and that reminded me of one of the main reasons that hiring by screening resumes is demonstrably less effective than hiring by giving work sample tests: many people lie on their resumes. The author of the interesting submitted blog post was of course posting a fake job ad, and several comments here on HN point out that many job ads may just be fronts for recruiters rather than postings by actual employers. Participants here gave examples (NUMEROUS examples) of job ads getting responses that don't appear at all to fit the job, but an employer also has to worry about "false positives," applicants who look like they fit the job but who are inflating their educational credentials or multiplying their years of actual work experience. The only way to know what an applicant can do is to test. Perhaps announcing up front that the hiring process includes actual testing of job-related skills MAY screen out some of the poseurs from even sending in their fake resumes (although the many shotgun applicants who don't even read the job ad closely will still be sending resumes all over the place only to waste the time of anyone who receives the resumes).
One way the author of the blog post could have demonstrated statistical acumen is by labeling his data presentations "Self-Reported Experience" rather than "True Experience" and "Self-Reported Credentials" rather than "Education." He has NO idea what the actual educational credentials or work experience (or other aspects of biography) of any of his applicants really are. He would have been aware of this point if he had taken a good statistics course in college, but alas good statistics classes are very rare in the United States.
http://statland.org/MAAFIXED.PDF
http://escholarship.org/uc/item/6hb3k0nz