Re:Reflections as the Internet Archive turns 25
by joe_the_user on Thursday July 22, 2021 @03:32AM
(#27916411)
Well, "complain on hn" has been a way to get stuff from Google. Maybe someone at archive will notice this thread.
Re:Reflections as the Internet Archive turns 25
by hidden-spyder on Thursday July 22, 2021 @01:19PM
(#27921649)
I'm curious. What changes has Google made due to complains on HN?
The question I have is how fast is the content removed after the domain name registration changes, i.e., is there is a window of time between the appearance of a new robots.txt and the next scheduled crawl, and if so, is it be possible to "rescue" the content, as ArchiveTeam would do, during that window, before it disappears.
If this is possible, there could be a service for monitoring changes to domain name registrations for sites that have large amounts of historical content. I would happily volunteer to set up such a service.