News for hackers. Stuff that matters.
Re:An Introduction to Compassionate Screen Scraping by storborg on Tuesday April 19, 2011 @04:15PM (#2464037)
The author makes some great suggestions, namely to cache heavily and throttle requests. However, they lost a lot of credibility for me with "screen scraper traffic should be indistinguishable from human traffic". Sorry, but that's BS--socially responsible scraping leaves control with the publisher. If the publisher doesn't want you scraping their content, you shouldn't try to fake a human in order to be able to.
Re:An Introduction to Compassionate Screen Scraping by dotBen on Tuesday April 19, 2011 @04:32PM (#2464090)
I see your point - however I read from it that the author was more referring to the load/level of activity on the server that your requests make should be indistinguishable from human traffic.

IE if the server's log files have 100's of requests from the same IP address in successive lines then that doesn't look like human behavior.

What would have been nice for a 'best practice' document would be to show how to set the HTTP AGENT string for the crawler so that it had an identifier, version number and some contact method.

Re:An Introduction to Compassionate Screen Scraping by helwr on Tuesday April 19, 2011 @04:43PM (#2464130)
Re:An Introduction to Compassionate Screen Scraping by dotBen on Tuesday April 19, 2011 @05:39PM (#2464342)
I'm confused - your url is for a thread about Java but this best practice document is python-orientated
Re:An Introduction to Compassionate Screen Scraping by njs12345 on Tuesday April 19, 2011 @07:08PM (#2464596)
It's a joke, look at the person asking the question :)