News for hackers. Stuff that matters.
Re:Ask HN: How do you guys stop scrapers from mirroring your... by gexla on Sunday January 16, 2011 @12:36AM (#2108780)
This is basically cloaking (showing your content one way to regular readers and showing it a different way to search engines,) which could get you slapped by Google. Not worth this risk IMO. However, maybe there is a way to do this so you don't have to cloak? Not sure, I have never looked into it.
Re:Ask HN: How do you guys stop scrapers from mirroring your... by evgen on Sunday January 16, 2011 @12:56AM (#2108801)
Just support Google's AJAX crawling standards and disallow the escaped_fragment requests from crawlers that are not on your whitelist.
Re:Ask HN: How do you guys stop scrapers from mirroring your... by ToastOpt on Sunday January 16, 2011 @01:50AM (#2108880)
... Isn't that exactly what the New York Times does with its required sign-on? You get prompted to sign in unless the referrer URL is google.
Re:Ask HN: How do you guys stop scrapers from mirroring your... by wslh on Sunday January 16, 2011 @09:05AM (#2109348)
I didn't say cloaking, you can show the same content in javascript and html. Show html for search engines.