News for hackers. Stuff that matters.
Re:The Internet Archive Telethon by JoshTriplett on Sunday December 20, 2015 @09:11AM (#10767057)
I think the archive.org crawler should respect robots.txt as it looked at the time of the crawl. As a well-behaved robot, archive.org's crawler should fetch and respect robots.txt each time it crawls. However, archive.org should not retroactively delete old content when the current site puts up a robots.txt.

(To answer your other question, the robots.txt standard already allows giving different instructions to different crawlers.)

There are no comments on this comment yet.