Re:Ask HN: How do you guys stop scrapers from mirroring your...
by wslh on Sunday January 16, 2011 @12:33AM
(#2108777)
i) Javascript: Very few crawlers can read your javascript code. Look at the source html of Google. Is pure Javascript!
ii) Accept crawling based on IPs.
iii) Use captchas.
iv) Use cookies and Measure the crawling speed.
v) Think that if someone wants to copy your content, they can just look at the cache of some search engine, they don't need to crawl you.
[ Parent
| on HN ]
Re:Ask HN: How do you guys stop scrapers from mirroring your...
by regularfry on Sunday January 16, 2011 @04:01AM
(#2109041)
Javascript isn't going to stop me from crawling a site if I want the content. Browser automation is simply too good.
Re:Ask HN: How do you guys stop scrapers from mirroring your...
by gexla on Sunday January 16, 2011 @12:36AM
(#2108780)
This is basically cloaking (showing your content one way to regular readers and showing it a different way to search engines,) which could get you slapped by Google. Not worth this risk IMO. However, maybe there is a way to do this so you don't have to cloak? Not sure, I have never looked into it.
Re:Ask HN: How do you guys stop scrapers from mirroring your...
by evgen on Sunday January 16, 2011 @12:56AM
(#2108801)
Just support Google's AJAX crawling standards and disallow the escaped_fragment requests from crawlers that are not on your whitelist.