News for hackers. Stuff that matters.
Re:The Internet Archive Telethon by anon4 on Sunday December 20, 2015 @05:52AM (#10766713)
robots.txt already lets you specify per-robot behaviour. You can trivially opt-out of crawling, but opt-in to archiving by explicitly allowing archive.org's bot and disallowing all other user agents.

There are no comments on this comment yet.