Re:How Google’s Web Crawler Bypasses Paywalls
by roywiggins on Saturday February 20, 2016 @12:57AM
(#11139097)
If those sites don't want spiders, they can just specify that in robots.txt, which Google honors, right?
From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.
From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.