Re:The Internet Archive is waging war on misinformation
by pishpash on Wednesday September 18, 2019 @06:00PM
(#21010732)
Does it also continue to archive while the 'do not archive' flag is on?
[ Parent
| on HN ]
Re:The Internet Archive is waging war on misinformation
by ent101 on Wednesday September 18, 2019 @10:40PM
(#21012739)
I'm not sure, but I do think they follow the standard guidelines, meaning they won't archive if the flag is on. This can be verified by going through the Heritrix crawler's source code.