News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by libeclipse on Friday April 21, 2017 @02:35AM (#14163520)
On the linked page, I see comments about ignoring the webmasters' wishes et al.

All I can say is f*ck that. It's a free and open internet. If you put content up on a public site, anyone has the right to go and look at it. Stop complaining when someone saves it.

And sure some people complain that scrapers slow down their site and that's why they use robots.txt, but really? Really? It's 2017 and your site is affected by that. I think you have bigger things to worry about.

Re:Robots.txt meant for search engines don’t work well for w... by bkor on Friday April 21, 2017 @06:11AM (#14164272)
> Really? It's 2017 and your site is affected by that.

That someone wants to use a robot to completely scrape an entire dynamic website is their goal. A site is not responsible to make that possible. One bot causes _way_ more traffic and CPU usage than just a normal visitor or 1000s of visitors.

Saying '2017' or anything else: meh.

Various network operators are pretty helpful. Sending abuse complaints regarding misbehaving bots has resulted in actions before. I've seen action being taken from universities, ISPs, etc. Though normally the bots are auto-blocked (on IP address or ranges; quite easy to script).

robots.txt is an established / de facto standard. Ignore it, be prepared to explain why. IMO pretty much any country have computer hacking laws which are vague enough that to consciously ignore such a standard can be seen as "invading".

A "not my problem" approach: I think you should really think a little bit more.

Re:Robots.txt meant for search engines don’t work well for w... by madshiva on Friday April 21, 2017 @07:04AM (#14164477)
I totally ignore it and my bot never get caught. If they catch me I will say that the script wasn't working correctly, but what you are saying is wrong, there's is NO LAW stating that /robots.txt must be obeyed. Therefore it's not my problem, I just don't follow your rule, but I have the choice too and you have the choice to block my IP too which I think is more harmful.

Also thanks for spreading bad information.

Re:Robots.txt meant for search engines don’t work well for w... by adventured on Friday April 21, 2017 @07:23AM (#14164562)
> there's is NO LAW stating that /robots.txt must be obeyed. Therefore it's not my problem

You're not wrong about robots.txt, you're wrong in a much more broad way. There is in fact an extremely dangerous law that could easily ensnare what you're talking about:

https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act

Re:Robots.txt meant for search engines don’t work well for w... by madshiva on Friday April 21, 2017 @08:55AM (#14165108)
I don't know if the CFAA apply to my country, I know moreover that we don't need to comply with DCMA.

I don't thing that browsing a web page and saving it's content it's the same than scamming people by doing fake online site. This is growing in our country and the local police don't have any rights.

If it's a global problem we need to have global rules, we can't have Chinese not respecting Authors' rights and in the other hand only blame local people it's stupid.

Specially when it's non-tech people that do the rules, they don't know tech therefore should not say anything about it.

EDIT: You can be mad at me and down vote, but what I say is true and relevant. There's not only US in the world, specially when there's other way than protecting your site behind a robots.txt

Re:Robots.txt meant for search engines don’t work well for w... by icebraining on Friday April 21, 2017 @10:05AM (#14165708)
We do have global rules: the Berne convention, which has been ratified by all 170 UN countries plus the Holy See and Niue, states that copyright is automatic and mostly universal, so any unauthorized copying is illegal. By having certain paths listed on robots.txt, the site are explicitly saying they don't authorize people to crawl them, so unless you have a license granting you permission, your legal position is probably iffy - CFAA or not.

Obviously some countries have a more lax enforcement than others, but don't be surprised if the US starts squeezing and one day you suddenly get a knock on the door.

Re:Robots.txt meant for search engines don’t work well for w... by LinuxBender on Friday April 21, 2017 @08:51AM (#14165070)
I agree that humans will do what humans can do and bots will do what bots can do. Laws are murky and I don't wish to donate to lawyers. I believe engineering solutions when possible is the answer.

Using simple conditional tests in haproxy, I stop most of the bots from crawling anything more than my root page, robots.txt and humans.txt. Anything else gets silently dropped and the bots will retry for a while then go away. I don't see anything in the logs beyond the root page and robots/humans.txt any more.

Re:Robots.txt meant for search engines don’t work well for w... by shaki-dora on Friday April 21, 2017 @11:49AM (#14166607)
Hey everyone, look & archive! This is where Jerome Renoux of Akamai announces that he doesn't believe in any morality beyond that codified in law, and how he will lie in court if you try to get him to behave decently.
Re:Robots.txt meant for search engines don’t work well for w... by LoSboccacc on Friday April 21, 2017 @05:05AM (#14164044)
they only ever needed to honor the robots.txt at the date of archival.

archive.org fucked up by making robots retroactive, if they used the archived robots.txt as a filter for a site at the relevant date, they'd have had the best of both worlds - respecting how sites work without losing how sites appeared at a date.

Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Friday April 21, 2017 @05:38AM (#14164165)
They'd be the flouting copyright laws, like Google do, but nonetheless tortuously. They're making a copy, which is already an infringement, distributing it seemingly against the owners express wishes is treated as a crime in some jurisdictions.
Re:Robots.txt meant for search engines don’t work well for w... by Retric on Friday April 21, 2017 @05:47AM (#14164199)
Robots.TXT is explicit permission to make a copy, otherwise crawling is meaningless and that is not nessisarily reversible. Like putting up a yard sale sign, then trying to get the poeple that show up yesterday arrested for trespassing.

What the archive can do after that point is a different issue, but they clearly can keep a copy. Further, someone else is using the domain they don't nessisarily have anything to do with the archived data.

Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Friday April 21, 2017 @07:24AM (#14164565)
Robots.txt is usually explicitly permission for a robot not to crawl. But a robot crawling your site and an archive, cache, or duplicate page are all different propositions.

Google and others have enhanced robots.txt to enable permission for crawling (allow, sitemap), meta tags can deny archiving and various means allow permission to be explicitly denied for caching.

To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal.

FWIW I disprove of this state of affairs and consider copyright to be hugely defective in these respects.

>but they clearly can keep a copy //

It's nuanced but permission to access a page =/= permission to keep a copy. Just as you have explicit permission to access a video on YouTube but in most jurisdictions will not have permission to download it for later (commercial) use.

Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @01:48PM (#14167664)
>To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal.

Right. And it's actually not a bad analogy as analogies go. Not having a sign doesn't make trespassing legal but if someone sometimes walks over a corner of your property and you go to the police to try to get him arrested, the first thing they'll probably ask you is if you have your property posted and/or if you bothered to ask him to stop. If the answer is no, they'll probably tell you to go away and do so and only come back if he ignores the sign.

Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:07AM (#14171972)
Yeah, you come down to breakfast and there's​ a random person sat on your sofa, you ring the police and they say "meh, you didn't put a sign up"?

The requirement to post a sign to make trespassing an actionable offence is a USA thing AIUI, it's not a UK thing at least, but copyright is almost universal and doesn't require even adding a (c) mark, it's automatic at the point of creation under the Berne Convention. Or in other words you've pushed your analogy too far and hit a marked legal difference between USA physical property law enforcement and international intellectual property law.

Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Saturday April 22, 2017 @09:08AM (#14172319)
Clearly entering a house is much different from the specific example I gave of walking across a corner of a property. Also, a number of number of European countries have various variants of right to roam that make trespassing a less actionable offense than in the US.

The more general principle is that if no harm is done and the individual/organization will just stop the action if you asked, the courts are often reluctant to get involved. There are exceptions of course, especially in the vein of making an example of someone to discourage others.

Re:Robots.txt meant for search engines don’t work well for w... by Retric on Friday April 21, 2017 @09:10AM (#14165220)
Not having robots.txt is having no sign. So if you have the file you are putting up a sign that says something.

Also, it's meaningless for a bot to get permission without having permission to make a copy. There are arguments around the number of copies, but the clear implication is at lest all the routers can make a copy.

Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @10:27AM (#14165874)
>So if you have the file you are putting up a sign that says something.

Not really. There are often default robots.txt files that the system just puts there in the course of building a default website.

The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.

Re:Robots.txt meant for search engines don’t work well for w... by Retric on Saturday April 22, 2017 @12:04AM (#14171071)
That might hold water if the robots.text where the default on every site ever used by a company, but change even a single bit on any of them and what's left is clearly your intent on all of them.

Further, setting up a physical device connected to a public IP is never default behavior so you are putting up the sign in either case. So, at best your argument is someone athorised to do something put up a sign by mistake saying something that was not intended, but your intent has little relevance at that point.

Worse, your argument is based on the assumption that nobody knew what was going on so even simple coursework mentioning robots.txt would demonstrate knowledge and thus intent through willful inaction.

Re:Robots.txt meant for search engines don’t work well for w... by matt4077 on Friday April 21, 2017 @11:56AM (#14166671)
IA has every right to scrape, save, and display. See US Code 108. Robots.txt is mostly meaningless in terms of the law, because it doesn't say anything about copying or distributing–only access, which is outside the scope of copyright.
Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:22AM (#14172001)
I think you mean 17-USC-108. It might surprise you to know that the web extend outside USA and that Fair Use and exemptions for libraries and archives (aside: where's the definition of those two terms in the USC that's being used here?) don't extend to the World. IA may be legal in USA but a whole lot of the websites of the world reside on servers in other countries.

There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?

Re:Robots.txt meant for search engines don’t work well for w... by reitanqild on Friday April 21, 2017 @06:30AM (#14164354)
Anyone knows why this was downvoted?
Re:Robots.txt meant for search engines don’t work well for w... by matt4077 on Friday April 21, 2017 @11:52AM (#14166633)
The law has specific exemptions for archival and search, which google and IA use. Do you really think they could do what they're doing when any one of millions of people could just take them to court at any time?
Re:Robots.txt meant for search engines don’t work well for w... by ghaff on Friday April 21, 2017 @12:26PM (#14166971)
Well, they have been sued, e.g. https://arstechnica.com/uncategorized/2006/08/7634/ and they settled.

But they're not a target that you're going to collect big from, as a non-profit archive/library they're sympathetic whether or not that gives them any special legal standing, and they'll basically take down your content past and present if you ask them to.

So it will almost certainly cost you money to sue them, you won't collect much in the best case, and you can get your content taken down in about as much time as it would take you to pick a lawyer out of the phone book.

Re:Robots.txt meant for search engines don’t work well for w... by pbhjpbhj on Saturday April 22, 2017 @07:24AM (#14172007)
So my website is on a server outside the USA, what exemption are they relying on?
Re:Robots.txt meant for search engines don’t work well for w... by blowski on Friday April 21, 2017 @04:30AM (#14163912)
What's your opinion on Google scraping your content and putting minimally attributed snippets?
Re:Robots.txt meant for search engines don’t work well for w... by hoschicz on Friday April 21, 2017 @08:23AM (#14164850)
"minimally attributed"?

On Google? Really?

Re:Robots.txt meant for search engines don’t work well for w... by fenwick67 on Friday April 21, 2017 @10:16AM (#14165788)
See this article for examples of how snippetsa don't always attribute correctly:

https://theoutline.com/post/1399/how-google-ate-celebritynet...

The author runs CelebrityNetWorth.com, which BusinessInsider cites, but the snippets cite BusinessInsider. So the user doesn't see the proper attribution.

Re:Robots.txt meant for search engines don’t work well for w... by chii on Friday April 21, 2017 @03:13AM (#14163653)
> scrapers slow down their site and that's why they use robots.txt

a poorly written scraper may really slow down your site, especially if it wasn't intended to be scrapped repeatedly. There should be something to be said about frequency which scrapers should follow (specified by the website owner via a robots.txt like spec).

But website owners cannot demand unreasonable frequencies (such as once a year!), and what constitutes unreasonable is up for debate.

Re:Robots.txt meant for search engines don’t work well for w... by eknkc on Friday April 21, 2017 @03:49AM (#14163771)
I don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.
Re:Robots.txt meant for search engines don’t work well for w... by davb on Friday April 21, 2017 @03:34AM (#14163710)
Additionally, if excessive scraping became an issue for my site I'd consider rate limiting client.
Re:Robots.txt meant for search engines don’t work well for w... by dingo_bat on Friday April 21, 2017 @04:21AM (#14163875)
> (specified by the website owner via a robots.txt like spec).

Nope, if a website wants such a restriction, it must enforce it. Robots.txt is a request. It's worthless.

Re:Robots.txt meant for search engines don’t work well for w... by bkor on Friday April 21, 2017 @06:13AM (#14164281)
If a robot misbehaves, it'll either be blocked or it'll go to the networks abuse section and that bot will be taken down. That a site possibly could have some kind of technical solution to this doesn't matter.
Re:Robots.txt meant for search engines don’t work well for w... by problems on Friday April 21, 2017 @11:25AM (#14166373)
Precisely - the solution here needs to be that the server blocks the robot - if it can differentiate it from other traffic that is. That's all well and good and that's the solution which should be used here. If you don't want to be archived, block the IP.
Re:Robots.txt meant for search engines don’t work well for w... by timClicks on Friday April 21, 2017 @03:38AM (#14163726)
The Crawl-delay directive is the de facto standard for this.
Re:Robots.txt meant for search engines don’t work well for w... by un-devmox on Friday April 21, 2017 @10:56AM (#14166119)
> If you put content up on a public site, anyone has the right to go and look at it.

Fine.

> Stop complaining when someone saves it.

Fine.

What you don't say is that it is fine to recreate and publish that content against the owner's wishes, especially when said content is copyrighted in one way or another. You're failing to see the whole picture from the content owner's point of view.

Re:Robots.txt meant for search engines don’t work well for w... by hyperdunc on Friday April 21, 2017 @01:22PM (#14167444)
Perhaps implicit in the comment you replied to is the idea that there's no such thing as a "content owner".
Re:Robots.txt meant for search engines don’t work well for w... by un-devmox on Friday April 21, 2017 @03:06PM (#14168385)
That's the debate I was trying to initiate!

For instance, is it OK to crawl a blog with explicit copyright, save that data, then publish it elsewhere?

Re:Robots.txt meant for search engines don’t work well for w... by Houshalter on Friday April 21, 2017 @12:59PM (#14167270)
Consider this very website, hacker news. Every single comment has it's own url. Which creates a huge number of redundant urls. There's no reason for archive sites to be scraping a separate copy of every single comment after it's indexed the thread. And it makes it harder to use google to search HN.
Re:Robots.txt meant for search engines don’t work well for w... by bigbugbag on Friday April 21, 2017 @02:27PM (#14168020)
Aw man you should have been there 10 years ago, this website hacker news had a url to access every single comment. And quite often some of the comment were more informative than the link poster, I've bookmarked quite a few of those myself. But with hacker news gone they're gone too, because archive.org failed: though they do have the data they broke the links and it is now inaccessible to me.
Re:Robots.txt meant for search engines don’t work well for w... by Houshalter on Friday April 21, 2017 @03:38PM (#14168712)
I see your point. Perhaps a better solution would be to get rid of permalinks to individual comments entirely and point to them with fragment identifiers.
Re:Robots.txt meant for search engines don’t work well for w... by tomjen3 on Friday April 21, 2017 @06:49AM (#14164422)
They also make the content available to the public, which is directly competing with the site owners.

Thats inviting lawsuits they can't win and expecting people to pay the bandwidth for it too.