Re:Removing Your Site from the Wayback Machine
by danso on Wednesday June 12, 2019 @09:57AM
(#20164649)
Besides the aforementioned gov/mil sites that archive.org announced a changed policy on, I noticed during the TurboTax controversy [0] that archive.org seems to have disobeyed robots.txt (2008) [1a,1b] and NOARCHIVE (2006) [2] for TurboTax as well. It wouldn't surprise me if Archive.org retroactively changed this because of the controversy, since they manually changed display settings during the Jo Reid [3] controversy. But the latter case involved a webmaster who set robots.txt to wipe out old pages, whereas Turbotax have had robots.txt + NOARCHIVE as a policy for years. I had assumed Archive.org wouldn't even originally store such pages during the crawl, but that seems not to be the case?
[0] https://news.ycombinator.com/item?id=19758126
[1] http://web.archive.org/web/20080512041438/http://turbotax.in...
http://web.archive.org/web/20080512004753/http://turbotax.in...
[2] https://twitter.com/dancow/status/1123762475244163078
[3] https://blog.archive.org/2018/04/24/addressing-recent-claims...