News for hackers. Stuff that matters.

A comment on Google introducing tool for under-18s to remove images of them from search. Parent comment

Re:Google introducing tool for under-18s to remove images of... by lstamour on Thursday August 19, 2021 @12:51PM (#28236197)
In defence of search engines, when content disappears from the web, it disappears from the search index within months. And robots.txt has been a thing since forever.

But I agree with the overall point, that archiving is distinct from indexing, and we need better mechanisms than copyright to take down content given many services terms allow content to be kept forever and/or make it hard to delete content uploaded by others. This might come down to free speech vs privacy, though.

[ Parent | on HN ]
Re:Google introducing tool for under-18s to remove images of... by Sanzig on Thursday August 19, 2021 @02:52PM (#28237747)
> And robots.txt has been a thing since forever.

Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files.

Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself.

I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look.

[1] http://wiki.archiveteam.org/index.php?title=Robots.txt

[ Parent | on HN ]
Re:Google introducing tool for under-18s to remove images of... by floatingatoll on Thursday August 19, 2021 @12:54PM (#28236247)
If I ever again post a website, I’m going to post it with basic auth, so that search engines won’t index it to begin with. Exposure doesn’t pay the bills, after all.
[ Parent | on HN ]