Re:Ask HN: Legality of Web Scraping User-Submitted Content?
by webwright on Wednesday August 06, 2008 @05:42PM
(#269041)
Google does it, as do vertical search companies like Indeed and SimplyHired, so it's certainly legal under some conditions. Taking only a snippet and linking back to the source is generally okay (as the original publisher benefits from traffic and SEO juice).
[ Parent
| on HN ]
Re:Ask HN: Legality of Web Scraping User-Submitted Content?
by jcl on Wednesday August 06, 2008 @06:03PM
(#269071)
I assume you're talking about Google re-serving cached content. I think Google's defense boils down to "Any site without a properly configured robots.txt file implicitly grants permission to be spidered, cached, and linked to.":
http://www.google.com/support/webmasters/bin/answer.py?answe...
I don't know if this "opt-out" strategy has been tested in court, but I wouldn't assume that wholesale copying of user content is analogous to Google's situation unless the users have some similarly accepted way of opting out of the copying.