Think of the potential alternatives. One is everything can be crawled with no way for the site owner to say "no, please don't crawl this."

The other major one is to legally bar all crawling without express permission.

The current de facto world - crawling is OK unless robots.txt says otherwise - is pretty nice. If we want that to be a legal defense in court ("You didn't put up a robots.txt, so my indexing was legal, so you can't sue me."), which seems useful, then the necessary flipside of that is that violating robots.txt exposes the crawler to liability. That's a tradeoff I'm perfectly willing to accept to allow the web, and necessary services such as indexers and crawlers, to work while still allowing publishers to have some reasonable control over their content distribution.

I seem to remember one of the writers at Search Engine Land presenting a nice description of the robots.txt request in contract negotiation terms. Something like this:

Archiver: "Are there any limits on what I can archive or index from your site?" (translation: GET /robots.txt)

Site: "Nope." (translation: 404 Not Found)

or

Site: "Yep, here they are." (translation: 200 OK followed by restrictions in robots.txt format)

So, by asking for /robots.txt, the crawler can be construed as asking permission to index, and the response setting up the terms of indexing. That seems like a really useful defense and sane compromise in this age of "indexing so you can drive search users to our content is copyright infringement."

[EDIT: fix formatting]

I guess the first alternative seems like the more sane one to me, as far as legality goes. Website-crawling policies don't seem like the kind of thing that rises to the level where it's worth involving courts and laws, so I'd leave it to technological mechanisms plus voluntary compliance with non-technological mechanisms. But I suppose I have a pretty high bar for what problems are severe enough to require a government solution.

And pragmatically, the vast majority of the non-robots.txt-respecting crawls I see are coming from countries that won't enforce such laws anyway, so enforcing them in western countries seems like a downside (more entanglement between the internet and various countries' national laws) with little upside (won't stop many crawls).

Don't be naive. People get sued over things that aren't codified in any law or part of the HTTP standard every day.

This is a bit of a legal grey area. The kind of thing that BigCos with expensive lawyers like to sue over. Would you really be willing to pay lawyers to settle it in court once and for all?

I think you misunderstood. I wasn't commenting on the likelihood, I was commenting on the justice of it. I'm not speaking as a practitioner of law (I am not one), I'm speaking as a subject of the law and it's equally my responsibility to respond to injustice.

Suing is not about what is lawful or not; suing is part of the civil court, not court of law. Anyone can sue anybody for any reason and the judge has the responsibility to decide if any wrongs were performed, and if they deserve compensation. So it isn't about what is lawful, it is about what bullshit they can convince the judge of, so getting a clued in judge is what you need.

The point you're missing here is that as there isn't a legal standard for this, robots.txt has arguably become a de-facto standard. It's widely understood that the things you mark as off-limits to crawlers in robots.txt should not be indexed or reproduced in automated fashion.

'Law' is not just written legislation handed down from legislatures and other 'sovereigns'. It's also made on the ground, by custom and consensus, and then recognized as wise and just by the courts in the case of disputes – at least in common-law jurisdictions like most of the English-speaking world:

http://en.wikipedia.org/wiki/Common_law

Robots.txt has a lot of custom and implied authority behind it by now – and as others have noted, some recognition by courts as well.