News for hackers. Stuff that matters.
Re:Internet Archive as a default host-of-record for startups by 1MachineElf on Tuesday December 21, 2021 @12:21PM (#29639586)
There is some public discussion about why IA does not strictly adhere to robots.txt:

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

Re:Internet Archive as a default host-of-record for startups by kodah on Tuesday December 21, 2021 @03:35PM (#29641790)
They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.
Re:Internet Archive as a default host-of-record for startups by BenjiWiebe on Tuesday December 21, 2021 @07:25PM (#29643920)
When those people die and quit paying for hosting, the information on their website doesn't magically become useless.

Perhaps other people still have a need for it.

Thank goodness the IA doesn't blindly obey robots.txt.