Quite. My robots.txt reads:

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /scratch/

This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.

At least, that's the way I used to roll.

The broader points being made in the article are that the value of information is determined by the visitor, and that the burden of keeping the site up and running should fall on the host.

By that reasoning, the presence of a ROBOTS.TXT signifies "damage" or "temporary madness", and should therefore be ignored.

Also, keep in mind where this article came from - the Archive Team are bloody-minded about preserving information.

If the data has value to the user, shouldn't the user be paying the host for the cost of making that data available (or perhaps considerably more, if it has a lot of value to the user)?

If the answer is "nothing", then I guess you've just told me that all your comments are useless, and I should ignore them.

Drivingmenuts says the user should pay for hosting data. HN is hosting his data, but he doesn't pay for it. Is there some error in my logic, here, some flaw in my conclusion?

They won't do a good job of that if they encourage too many people to make IP blocks of their machines.

  User-agent: * 
  Disallow: /

1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/

That may be so in a sense, unless the visitor has no means of making the call -- or does so poorly. I promise you that every robots.txt ignoring bot makes terrible judgments in that regard. (I'm not trying to overstate anything here; those that abide it make, on average, slightly less terrible judgments.)

But it's not just the visitor who gets to make judgment calls. The value of serving those pages is something the host can decide. Belligerent drunks who abuse the staff aren't allowed in the coffee shop, and if they do it often enough they're not allowed back in when they sober up either.