Re:Wayback machine gets a facelift, new features
by Cameron_D on Monday November 04, 2013 @07:16PM
(#6672157)
The worst part is when a site is stored in the wayback, then the domain expires and the new owner (or squatter) has a robots.txt that blocks everything, then all the old content becomes inaccessible.
[ Parent
| on HN ]
Re:Wayback machine gets a facelift, new features
by daveid on Tuesday November 05, 2013 @03:21AM
(#6673890)
They should store a history of WHOIS data for the site's domain and make separate archives for when the owner changes, I think. Also, why did anyone think that applying robots.txt retroactively is a good idea? :/