Re:Post Ghost Shutdown: An Open Letter to Twitter
by homero on Saturday July 09, 2016 @01:27AM
(#12060249)
Archive projects usually listen to robots.txt
[ Parent
| on HN ]
Re:Post Ghost Shutdown: An Open Letter to Twitter
by unimpressive on Saturday July 09, 2016 @01:29AM
(#12060255)
ArchiveTeam on the other hand is a bit more aggressive:
http://www.archiveteam.org/index.php?title=Main_Page
Though they generally do last minute emergency backups of closing sites, so this probably isn't a job for them.