Re:Google introducing tool for under-18s to remove images of...
by Sanzig on Thursday August 19, 2021 @02:52PM
(#28237747)
> And robots.txt has been a thing since forever.
Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files.
Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself.
I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look.
Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files.
Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself.
I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look.
[1] http://wiki.archiveteam.org/index.php?title=Robots.txt