Preventing Site Scraping

Re:Preventing Site Scraping by jwdunne on Thursday May 17, 2012 @05:18PM (#3989151)

A less aggressive approach I've encountered is to insert links to other pages of your website with full URLs (http://www.example.com/page.html over just /page.html). Usually, a scraper will copy the links too. This should then make it obvious that the content's been scraped.

This could become a nightmare to maintain if you don't automate it. It'd be trivial to automate on a CMS. I know WordPress has loads of plugins for exactly this. I don't think I've come across something that can do this for static websites though, which make up for the brunt of the websites I maintain.

[ Parent | on HN ]

Re:Preventing Site Scraping by epoxyhockey on Thursday May 17, 2012 @05:35PM (#3989231)

wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

[ Parent | on HN ]

Re:Preventing Site Scraping by chc on Thursday May 17, 2012 @05:48PM (#3989296)

If your data matters at all to the scraper, this won't present much of an obstacle. Fixing up internal links is really easy. Your average script kiddie could probably figure out how to work around this in a matter of hours. Like so many "tricks", it sounds like more trouble for you than for them.

[ Parent | on HN ]