Ask HN: Best way to block an EC2-hosted scraper?

Re:Ask HN: Best way to block an EC2-hosted scraper? by a5seo on Monday January 10, 2011 @11:39AM (#2088905)

I like it, but you need to do this in a way that you append a known/deterministic string so you can monitor THAT one in Google Alerts.

I don't care whether the unique strings collide (although I love the idea of being able to tie the IP address of the crawler to the page on which the content was published for maximum evidence collection), I just want Google Alerts to find them reliably AND I don't want the spammer to catch on and be able to easily remove the unique strings before I catch him.

The only way for him to find my little landmines (to strip them) would be to read through every piece of content he scraped from me.

[ Parent | on HN ]

Re:Ask HN: Best way to block an EC2-hosted scraper? by ay on Saturday January 15, 2011 @08:01PM (#2108274)

By the way - thinking of all this - why would not you turn this scraper into a free advertisement for you ? If you can detect when it crawls your site, just insert randomly the backlinks to your site into the content you give out to them.

This way you will want that they steal more from you :-)

(how to put these backlinks in a way that would be difficult to remove - is another story. But if you use other links, e.g. tinyurl, in your material - then you could probably use that :-) Still fairly simple to remove but would require more work from them.)

[ Parent | on HN ]