News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @10:27AM (#14165874)
>So if you have the file you are putting up a sign that says something.

Not really. There are often default robots.txt files that the system just puts there in the course of building a default website.

The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.

Re:Robots.txt meant for search engines don’t work well for w... by Retric on Saturday April 22, 2017 @12:04AM (#14171071)
That might hold water if the robots.text where the default on every site ever used by a company, but change even a single bit on any of them and what's left is clearly your intent on all of them.

Further, setting up a physical device connected to a public IP is never default behavior so you are putting up the sign in either case. So, at best your argument is someone athorised to do something put up a sign by mistake saying something that was not intended, but your intent has little relevance at that point.

Worse, your argument is based on the assumption that nobody knew what was going on so even simple coursework mentioning robots.txt would demonstrate knowledge and thus intent through willful inaction.

Re:Robots.txt meant for search engines don’t work well for w... by matt4077 on Friday April 21, 2017 @11:56AM (#14166671)
IA has every right to scrape, save, and display. See US Code 108. Robots.txt is mostly meaningless in terms of the law, because it doesn't say anything about copying or distributing–only access, which is outside the scope of copyright.
Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:22AM (#14172001)
I think you mean 17-USC-108. It might surprise you to know that the web extend outside USA and that Fair Use and exemptions for libraries and archives (aside: where's the definition of those two terms in the USC that's being used here?) don't extend to the World. IA may be legal in USA but a whole lot of the websites of the world reside on servers in other countries.

There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?