Re:I've mothballed my blog
by JoshTriplett on Monday September 30, 2013 @04:25PM
(#6472128)
Yeah, they really ought to work from the version of robots.txt archived along with a given snapshot. If the legal owner of the content wants it taken down, they can submit an explicit request for that.
Which sucks in an archival perspective.