""To remove your site from the Wayback Machine, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt).
The robots.txt file will do two things:
1. It will remove documents from your domain from the Wayback Machine.
2. It will tell us not to crawl your site in the future.
To exclude the Internet Archive’s crawler (and remove documents from the Wayback Machine) while allowing all other robots to crawl your site, your robots.txt file should say:
User-agent: ia_archiver
Disallow: /""
Their current documentation no longer says that they stop displaying old archives automatically in the presence of an ia_archiver Disallow directive, but I have not experimented about whether they still actually do this anyway.
Just guessing wildly about similar issues, I know that news organizations which have a publicly documented "unpublish" policy tend to get that policy used aggressively by reputation management firms and the like.
""To remove your site from the Wayback Machine, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt).
The robots.txt file will do two things:
1. It will remove documents from your domain from the Wayback Machine. 2. It will tell us not to crawl your site in the future.
To exclude the Internet Archive’s crawler (and remove documents from the Wayback Machine) while allowing all other robots to crawl your site, your robots.txt file should say:
User-agent: ia_archiver Disallow: /""
Their current documentation no longer says that they stop displaying old archives automatically in the presence of an ia_archiver Disallow directive, but I have not experimented about whether they still actually do this anyway.
Just guessing wildly about similar issues, I know that news organizations which have a publicly documented "unpublish" policy tend to get that policy used aggressively by reputation management firms and the like.