Re:The Internet Archive Telethon
by anon4 on Sunday December 20, 2015 @05:52AM
(#10766713)
robots.txt already lets you specify per-robot behaviour. You can trivially opt-out of crawling, but opt-in to archiving by explicitly allowing archive.org's bot and disallowing all other user agents.
[ Parent
| on HN ]
There are no comments on this comment yet.