News for hackers. Stuff that matters.
Re:Removing Your Site from the Wayback Machine by danso on Wednesday June 12, 2019 @09:57AM (#20164649)
Besides the aforementioned gov/mil sites that archive.org announced a changed policy on, I noticed during the TurboTax controversy [0] that archive.org seems to have disobeyed robots.txt (2008) [1a,1b] and NOARCHIVE (2006) [2] for TurboTax as well. It wouldn't surprise me if Archive.org retroactively changed this because of the controversy, since they manually changed display settings during the Jo Reid [3] controversy. But the latter case involved a webmaster who set robots.txt to wipe out old pages, whereas Turbotax have had robots.txt + NOARCHIVE as a policy for years. I had assumed Archive.org wouldn't even originally store such pages during the crawl, but that seems not to be the case?

[0] https://news.ycombinator.com/item?id=19758126

[1] http://web.archive.org/web/20080512041438/http://turbotax.in...

http://web.archive.org/web/20080512004753/http://turbotax.in...

[2] https://twitter.com/dancow/status/1123762475244163078

[3] https://blog.archive.org/2018/04/24/addressing-recent-claims...

There are no comments on this comment yet.