News for hackers. Stuff that matters.
Re:Internet Archive as a default host-of-record for startups by kodah on Tuesday December 21, 2021 @03:40PM (#29641844)
I mean even honoring the opt-out of robots.txt would be fantastic. As another commenter pointed out they willfully ignore it: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

It's fairly unethical.

There are no comments on this comment yet.