News for hackers. Stuff that matters.
Re:Reflections as the Internet Archive turns 25 by 1vuio0pswjnm7 on Thursday July 22, 2021 @02:37AM (#27916107)
This is an old problem I could have sworn there were promises they were going to change their procedures.

The question I have is how fast is the content removed after the domain name registration changes, i.e., is there is a window of time between the appearance of a new robots.txt and the next scheduled crawl, and if so, is it be possible to "rescue" the content, as ArchiveTeam would do, during that window, before it disappears.

If this is possible, there could be a service for monitoring changes to domain name registrations for sites that have large amounts of historical content. I would happily volunteer to set up such a service.

Re:Reflections as the Internet Archive turns 25 by joe_the_user on Thursday July 22, 2021 @03:32AM (#27916411)
Well, "complain on hn" has been a way to get stuff from Google. Maybe someone at archive will notice this thread.
Re:Reflections as the Internet Archive turns 25 by hidden-spyder on Thursday July 22, 2021 @01:19PM (#27921649)
I'm curious. What changes has Google made due to complains on HN?