Any process that does not cause a cold high-quality lead to come into contact with an engaged human being almost automatically is broken.
And incidentally, cutesy programming quizzes are often a very good filter. In my last job, I tried recruiting with one after speaking to too many people who couldn't handle fizzbuzz. The exact quote was "Instructions on where to send your resume are contained in our application (you'll need to decrypt them first)." http://news.ycombinator.com/item?id=1659735
It dramatically increased the number of people who wrote to me - I actually got quite a few responses of "neat puzzle, not looking for a job, did I get it right?" And by virtue of not giving out an unencrypted email address, I didn't talk to anyone who couldn't code.
A false positive results in a candidate who can e.g. write a kernel memory allocator walking from your process because you're too much of a pain to deal with.
Too many interviews, or losing the best possible candidates. I know which one I prefer --- especially because I know when I'm getting too many interviews, and can tune other things (most notably: where I post about jobs) to adjust that. I can't know who I'm losing as a result of my recruiting process being cumbersome.
The other thing that jumps out at me is that instead of a cute challenge, it's probably more useful to provide a bug in 50 lines of code that you expect to "jump out" effortlessly for the qualified candidates, while less qualified might struggle a little.
This is trivially true of any hiring method - you might scare away a few good people. The right question to ask is "what are the proportoins?"
If I can scare away 100% of the chaff and 1% of the wheat, it's probably a win. I can simply hire anyone who survives. If the ratio is less stark, it may or may not be a win, depending on my costs of hiring, how badly I need someone, etc.
I'm not saying everyone who uses this tactic is doing it right, merely that it's not a bad idea in all cases.