Re:Matt Cutts is looking for scraper sites
by ezequiel-garzon on Friday February 28, 2014 @09:33AM
(#7318836)
The de facto standard robots.txt is pretty likely to be respected by Google, so it's fairly easy to stop their scraping your site. Yes, it is opt-out, bit I'd expect it to be.
It may be quite frustrating for an upstart to be denied access while Google is explicitly allowed, but that's another matter.
It may be quite frustrating for an upstart to be denied access while Google is explicitly allowed, but that's another matter.