News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by matt4077 on Friday April 21, 2017 @11:52AM (#14166633)
The law has specific exemptions for archival and search, which google and IA use. Do you really think they could do what they're doing when any one of millions of people could just take them to court at any time?
Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @12:26PM (#14166971)
Well, they have been sued, e.g. https://arstechnica.com/uncategorized/2006/08/7634/ and they settled.

But they're not a target that you're going to collect big from, as a non-profit archive/library they're sympathetic whether or not that gives them any special legal standing, and they'll basically take down your content past and present if you ask them to.

So it will almost certainly cost you money to sue them, you won't collect much in the best case, and you can get your content taken down in about as much time as it would take you to pick a lawyer out of the phone book.

Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:24AM (#14172007)
So my website is on a server outside the USA, what exemption are they relying on?