News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Friday April 21, 2017 @07:24AM (#14164565)
Robots.txt is usually explicitly permission for a robot not to crawl. But a robot crawling your site and an archive, cache, or duplicate page are all different propositions.

Google and others have enhanced robots.txt to enable permission for crawling (allow, sitemap), meta tags can deny archiving and various means allow permission to be explicitly denied for caching.

To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal.

FWIW I disprove of this state of affairs and consider copyright to be hugely defective in these respects.

>but they clearly can keep a copy //

It's nuanced but permission to access a page =/= permission to keep a copy. Just as you have explicit permission to access a video on YouTube but in most jurisdictions will not have permission to download it for later (commercial) use.

Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @01:48PM (#14167664)
>To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal.

Right. And it's actually not a bad analogy as analogies go. Not having a sign doesn't make trespassing legal but if someone sometimes walks over a corner of your property and you go to the police to try to get him arrested, the first thing they'll probably ask you is if you have your property posted and/or if you bothered to ask him to stop. If the answer is no, they'll probably tell you to go away and do so and only come back if he ignores the sign.

Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:07AM (#14171972)
Yeah, you come down to breakfast and there's​ a random person sat on your sofa, you ring the police and they say "meh, you didn't put a sign up"?

The requirement to post a sign to make trespassing an actionable offence is a USA thing AIUI, it's not a UK thing at least, but copyright is almost universal and doesn't require even adding a (c) mark, it's automatic at the point of creation under the Berne Convention. Or in other words you've pushed your analogy too far and hit a marked legal difference between USA physical property law enforcement and international intellectual property law.

Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Saturday April 22, 2017 @09:08AM (#14172319)
Clearly entering a house is much different from the specific example I gave of walking across a corner of a property. Also, a number of number of European countries have various variants of right to roam that make trespassing a less actionable offense than in the US.

The more general principle is that if no harm is done and the individual/organization will just stop the action if you asked, the courts are often reluctant to get involved. There are exceptions of course, especially in the vein of making an example of someone to discourage others.

Re:Robots.txt meant for search engines don’t work well for w... by Retric on Friday April 21, 2017 @09:10AM (#14165220)
Not having robots.txt is having no sign. So if you have the file you are putting up a sign that says something.

Also, it's meaningless for a bot to get permission without having permission to make a copy. There are arguments around the number of copies, but the clear implication is at lest all the routers can make a copy.

Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @10:27AM (#14165874)
>So if you have the file you are putting up a sign that says something.

Not really. There are often default robots.txt files that the system just puts there in the course of building a default website.

The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.

Re:Robots.txt meant for search engines don’t work well for w... by Retric on Saturday April 22, 2017 @12:04AM (#14171071)
That might hold water if the robots.text where the default on every site ever used by a company, but change even a single bit on any of them and what's left is clearly your intent on all of them.

Further, setting up a physical device connected to a public IP is never default behavior so you are putting up the sign in either case. So, at best your argument is someone athorised to do something put up a sign by mistake saying something that was not intended, but your intent has little relevance at that point.

Worse, your argument is based on the assumption that nobody knew what was going on so even simple coursework mentioning robots.txt would demonstrate knowledge and thus intent through willful inaction.

Re:Robots.txt meant for search engines don’t work well for w... by matt4077 on Friday April 21, 2017 @11:56AM (#14166671)
IA has every right to scrape, save, and display. See US Code 108. Robots.txt is mostly meaningless in terms of the law, because it doesn't say anything about copying or distributing–only access, which is outside the scope of copyright.
Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:22AM (#14172001)
I think you mean 17-USC-108. It might surprise you to know that the web extend outside USA and that Fair Use and exemptions for libraries and archives (aside: where's the definition of those two terms in the USC that's being used here?) don't extend to the World. IA may be legal in USA but a whole lot of the websites of the world reside on servers in other countries.

There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?