News for hackers. Stuff that matters.
Re:Internet Archive as a default host-of-record for startups by pseudalopex on Sunday January 02, 2022 @08:32PM (#29776275)
Respecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.

There are no comments on this comment yet.