News for hackers. Stuff that matters.
Re:Preventing Site Scraping by fleitz on Thursday May 17, 2012 @08:59PM (#3989895)
Yeah as a scraper I'd say that at most all these suggestions would do is make me turn to selenium/greasemonkey instead of mechanize/wget/httrack. Selenium is the bomb when people try to get fancy preventing scraping, how exactly are they supposed to detect the difference between a browser and a browser?

Getting banned is not a big deal, plenty of IPs & proxies out there. EC2 is your best friend as you can automate the IP recycling. Even Facebook/Twitter accounts are almost free.

Even the randomization wouldn't be particularly difficult to circumvent just save the page and then use a genetic algorithm with tunable parameters for the randomization, select the parameters that yield the most/best records.

What I'd actually fear is a system that just silently corrupted the records once scraping was detected, especially if it was intermittent, eg. 10-75% of records on a page are bogus and only every few pages. Or they started displaying the records as images (but I'm guessing they want Google juice)

There are no comments on this comment yet.