News for hackers. Stuff that matters.
Re:How Google’s Web Crawler Bypasses Paywalls by roywiggins on Saturday February 20, 2016 @12:57AM (#11139097)
If those sites don't want spiders, they can just specify that in robots.txt, which Google honors, right?

From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.

There are no comments on this comment yet.