Re:Internet Archive as a default host-of-record for startups
by pseudalopex on Sunday January 02, 2022 @08:32PM
(#29776275)
Respecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.
Their documentation about it is rather crap right now, but its in several of their FAQs about it