The end result is often that the best people just walk on by. Honesty in the hiring process cuts both ways.
This is trivially true of any hiring method - you might scare away a few good people. The right question to ask is "what are the proportoins?"
If I can scare away 100% of the chaff and 1% of the wheat, it's probably a win. I can simply hire anyone who survives. If the ratio is less stark, it may or may not be a win, depending on my costs of hiring, how badly I need someone, etc.
I'm not saying everyone who uses this tactic is doing it right, merely that it's not a bad idea in all cases.
Any process that does not cause a cold high-quality lead to come into contact with an engaged human being almost automatically is broken.
And incidentally, cutesy programming quizzes are often a very good filter. In my last job, I tried recruiting with one after speaking to too many people who couldn't handle fizzbuzz. The exact quote was "Instructions on where to send your resume are contained in our application (you'll need to decrypt them first)." http://news.ycombinator.com/item?id=1659735
It dramatically increased the number of people who wrote to me - I actually got quite a few responses of "neat puzzle, not looking for a job, did I get it right?" And by virtue of not giving out an unencrypted email address, I didn't talk to anyone who couldn't code.
A false positive results in a candidate who can e.g. write a kernel memory allocator walking from your process because you're too much of a pain to deal with.
Too many interviews, or losing the best possible candidates. I know which one I prefer --- especially because I know when I'm getting too many interviews, and can tune other things (most notably: where I post about jobs) to adjust that. I can't know who I'm losing as a result of my recruiting process being cumbersome.
The other thing that jumps out at me is that instead of a cute challenge, it's probably more useful to provide a bug in 50 lines of code that you expect to "jump out" effortlessly for the qualified candidates, while less qualified might struggle a little.
The solution to this is, again, have an outreach process that gets you Glengarry Leads, at a much lower volume. Where do your best candidates congregate? What are they most interested in? What do they like to talk about? What do they have opinions about? Think about those questions and then, for God's sake, don't just pick a job board based on them.
I really think the problem here is how people do outreach to candidates. I feel like too many companies haven't figured out that recruiting is a marketing problem, just like software sales. Companies get all kinds of creative addressing marketing problems --- but when it comes to hiring, they plug "reqs" into "job boards" and then try to "deflect" the "pretenders".
Calibrate your outreach so that you can safely pay individual attention to all candidates. Don't post reqs to Craigslist or Monster or Dice or whatever your HR person --- in fact don't let your HR person near recruiting --- thinks is the place you're supposed to post ads. You will still be giving "early no" responses to lots and lots of candidates, but it'll be manageable, and it'll be very unlikely for your process to cost you talented candidates.