A subpeona was already issued and this information could have still been extracted from the Internet Archive directly, just at a higher technological hurdle. If the information was still on the plaintiff's servers they could have just extracted it from there as well (or any backups that they might have had laying around), since it was deleted they needed other ways to recover it.

robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).