For a competent developer, a "cover letter brainteaser" that takes an hour to do is essentially equivalent to a $100+ application fee (in terms of opportunity cost).
Probably pretty low. But it's not designed to have a high conversion rate. It's really just to scare off pretenders and the under-qualified. I like to solve "Cover Letter Problems" for fun, but I've never bothered to apply to a company that had them. I don't think that means they're a failure.
The applicants themselves probably benefit the most as it gives them a window into the company's dev culture. Cutesy "gotcha!" problems that are abstract and removed too far from coding are generally a warning sign that it's a shop full of people who may be smart but are perhaps mediocre developers and/or the shop has non-technical management. On the flip side, problems that explore an applicants coding style and thought process tends to be an indicator the shop wants actual coders who can get shit done in an efficient manner, and that they care about things like Code quality, their tool chain, and care just enough about process to achieve repeatable results, but not so much as to become dogmatic.
I really think the problem here is how people do outreach to candidates. I feel like too many companies haven't figured out that recruiting is a marketing problem, just like software sales. Companies get all kinds of creative addressing marketing problems --- but when it comes to hiring, they plug "reqs" into "job boards" and then try to "deflect" the "pretenders".
Calibrate your outreach so that you can safely pay individual attention to all candidates. Don't post reqs to Craigslist or Monster or Dice or whatever your HR person --- in fact don't let your HR person near recruiting --- thinks is the place you're supposed to post ads. You will still be giving "early no" responses to lots and lots of candidates, but it'll be manageable, and it'll be very unlikely for your process to cost you talented candidates.
The end result is often that the best people just walk on by. Honesty in the hiring process cuts both ways.
This is trivially true of any hiring method - you might scare away a few good people. The right question to ask is "what are the proportoins?"
If I can scare away 100% of the chaff and 1% of the wheat, it's probably a win. I can simply hire anyone who survives. If the ratio is less stark, it may or may not be a win, depending on my costs of hiring, how badly I need someone, etc.
I'm not saying everyone who uses this tactic is doing it right, merely that it's not a bad idea in all cases.
Any process that does not cause a cold high-quality lead to come into contact with an engaged human being almost automatically is broken.
And incidentally, cutesy programming quizzes are often a very good filter. In my last job, I tried recruiting with one after speaking to too many people who couldn't handle fizzbuzz. The exact quote was "Instructions on where to send your resume are contained in our application (you'll need to decrypt them first)." http://news.ycombinator.com/item?id=1659735
It dramatically increased the number of people who wrote to me - I actually got quite a few responses of "neat puzzle, not looking for a job, did I get it right?" And by virtue of not giving out an unencrypted email address, I didn't talk to anyone who couldn't code.
A false positive results in a candidate who can e.g. write a kernel memory allocator walking from your process because you're too much of a pain to deal with.
Too many interviews, or losing the best possible candidates. I know which one I prefer --- especially because I know when I'm getting too many interviews, and can tune other things (most notably: where I post about jobs) to adjust that. I can't know who I'm losing as a result of my recruiting process being cumbersome.
The other thing that jumps out at me is that instead of a cute challenge, it's probably more useful to provide a bug in 50 lines of code that you expect to "jump out" effortlessly for the qualified candidates, while less qualified might struggle a little.
The solution to this is, again, have an outreach process that gets you Glengarry Leads, at a much lower volume. Where do your best candidates congregate? What are they most interested in? What do they like to talk about? What do they have opinions about? Think about those questions and then, for God's sake, don't just pick a job board based on them.
In regard to the change in hiring practices, "social presence resumes" are just a passing trend. It may result in better candidates in the short term, but people will learn to pad these new resumes and it won't be much more effective than paper resumes.
The best hiring practice will still be tapping your employees' social network and hunting down talented people who aren't happy in their current positions. Anything else is usually a shot in the dark.
1) Union Square Ventures is a leading name in this space - everyone knows Fred Wilson's blog. On top of that, they only have three "staff" listed on their site. USV can ask would-be employees to fly up to NYC to deliver resumes in person and they'll still get plenty of applicants.
2) The job openings in the article that focus on surveys and personality tests - supposedly instead of resumes - are clearly unskilled labor (e.g. social media intern). This is exactly as it was when I applied to work as a waiter at Olive Garden in 2000.
3) IGN's coding challenges are similarly suspect as a representatitve example because game dev is a glamor industry with a surfeit of naifs lining up for these underpaid jobs. Prized developers aren't going to participate in many coding-contest-as-hiring-lottery situations unless they're just really interested in the problem. Ask a YC company who's ever posted a hiring puzzle: what percentage of your correct submissions were from people who didn't even want the job?
The tactics cited above wrongheadedly invert the recruiting plan tptacek uses at Matasano: Start from the assumption that you want the best people possible and that those people will have plenty of great alternatives to working for you. Do your best to get your company's name in front of those people and to convince them that they'd have a rewarding time if they joined your team.
I trust Thomas's experience in identifying and recruiting talent far more than I do a career WSJ reporter's [1] trend spotting.
[1] http://www.linkedin.com/pub/rachel-emma-silverman/31/331/507