In 2009 a company called localbacon.com tried just that experiment: http://techcrunch.com/2009/09/14/tc50-job-seekers-pay-localb...
It seems that the experiment wasn't a great success, at least as a job board business model, because the company shifted course after a relaunch six months later: http://techcrunch.com/2010/03/22/jibe-localbacon-relaunch/
I've seen a number of posts on HN from developers saying, "I won't apply for a job if they require a {cover letter, resume, code sample}." Depending on your point of view, adding hurdles could either remove self entitled prima donnas or all of the code ninjas. The risk is that you create a selection bias the wrong direction.
Adding hurdles will definitely reduce the number of applicants, but what you'll find is that if you ask for X, Y, Z in the ad most people will still send you whatever they were going to send you before (most likely a resume and coverletter). You then know they didn't read the ad and thus should probably be deleted.
This is why lots of job ads on craigslist say: "write XYZ in the email title" buried in the text to filter for those people who aren't even reading the ad.
While this would open up possible fraud with disingenuous job postings with no intent of being filled, would something like charging $1 to apply to a position improve the situation? After all, this is the approach that Universities take when accepting applicants.