Re:Ask HN: How do you guys stop scrapers from mirroring your...
by guan on Sunday January 16, 2011 @02:29AM
(#2108928)
You could do a reverse lookup on crawler IP addresses to make sure that they match the User Agent. Google PTR records will end in google.com or googlebot.com. Maybe compare with the list at http://chceme.info/ips/
You probably can’t prevent this altogether, but you can probably do a lot to raise the cost of this type of crawling. The exception might be if there are people out to copy your content specifically and will actually tailor their crawlers to your countermeasures.
You probably can’t prevent this altogether, but you can probably do a lot to raise the cost of this type of crawling. The exception might be if there are people out to copy your content specifically and will actually tailor their crawlers to your countermeasures.