[1]: http://www.american-justice.org/upload/page/123/69/docket-18...
[2]: https://tonybox.net/tmp/ia_decl_pacer.pdf (downloaded from PACER)
How is this not indirect self-incrimination?
But more to the point, archive.org already has a copy of what you previously made public, and there's no reason to burden them with the expense of manually retrieving it as they're not party to the case, and it would be wasteful of the court's resources and archive.org's time to make them sue you for the expense involved.
Here's a more detail breakdown of how the 5th applies in a civil litigation context: http://www.litigationandtrial.com/2013/04/articles/attorney/...
The bottom line is that the 5th can help you stay out of prison, but you can't use it just to avoid losing a dispute.
Things you have in your mind or - in some cases - in an extension of your mind (you're secretly actually guilty, you know where incriminating evidence is, or a safe combination or password that you haven't forgotten) cannot be compelled.
Things you possess (the contents of your home or gun safe, the thumb drive clenched in your fist, the robots.txt file on your server) can be compelled with a search warrant or subpoena. They are your things, and they could incriminate you, but that's not what the 5th Amendment is for.
Laconic.net is a resource I have enjoyed. Here is the 5th amendment passage: http://lawcomic.net/guide/?p=2470 or go back for a history lesson.
The modern Miranda passage starts here: http://lawcomic.net/guide/?p=2589
robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).
Subpoena?
The only thing that the judge ruled in this case is that it CAN be used.
At least they cannot add content.
And it is easy to know if they may have hidden something. Since the robots.txt should be public. I hope. May be a judge can make them unhide. I don't know.
This is quite disappointing (though probably wise from a legal standpoint). It makes it less useful if a party in a lawsuit can retroactively hide evidence of wrongdoing, and thereby deny access to evidence.
It's also a problem if a domain name changes ownership and the new owner suddenly adds a restrictive robots.txt - the old content will no longer be accessible even though the current owner has no claim to it.