Re:Screenshots Forever and Ever Until You Can’t Stand it
by jleader on Wednesday August 20, 2014 @02:56AM
(#8201343)
A few years ago I talked to an IA engineer, who said they were planning on dealing with this by not crawling sites whose nameservers were known to point to a domain parking company. The idea was that if they never retrieved the robots.txt, they wouldn't retroactively apply it. I don't know if that filtering out of parking nameservers ever happened, and it wouldn't help for parked domains whose robots.txt they'd already retrieved, but but it would help with domains that lapse in the future.
[ Parent
| on HN ]
Re:Screenshots Forever and Ever Until You Can’t Stand it
by StavrosK on Wednesday August 20, 2014 @05:32AM
(#8201660)
But why retroactively remove the data? The original owner was fine with holding it, why should the snapshot be deleted because a completely different person wants his completely different website to not be crawled?
Re:Screenshots Forever and Ever Until You Can’t Stand it
by bjt on Wednesday August 20, 2014 @12:50PM
(#8203128)
It's hard for a bot to understand concepts of 'owner' and 'completely different person' based on the data they have available. Companies can use this robots.txt feature to un-index old marketing content after a re-branding, for example. Or after an acquisition.
Re:Screenshots Forever and Ever Until You Can’t Stand it
by StavrosK on Wednesday August 20, 2014 @12:55PM
(#8203159)
Sure, but, surely, the bot has timestamps saying "robots.txt allowed me to keep these documents last time I spidered them". Why do they have to be retroactively removed? robots.txt only disallows spidering, it doesn't mandate that you should delete all the data you've already spidered.
Re:Screenshots Forever and Ever Until You Can’t Stand it
by db48x on Thursday August 21, 2014 @01:31AM
(#8206030)
Because most of the problems come from people who want to hide old material that they didn't realize was being indexed. The automatic behavior is simple and easy to implement, and doesn't require any human intervention.
Re:Screenshots Forever and Ever Until You Can’t Stand it
by im3w1l on Wednesday August 20, 2014 @08:16AM
(#8201997)
Imo, a better approach than nuking old data, could be to keep the data, but not show it.
Re:Screenshots Forever and Ever Until You Can’t Stand it
by psykovsky on Wednesday August 20, 2014 @08:55AM
(#8202115)
I may be mistaken, but I think I read somewhere that's exactly what they do. But don't take my word for it.
Re:Screenshots Forever and Ever Until You Can’t Stand it
by ersii on Wednesday August 20, 2014 @12:47PM
(#8203116)
You are not mistaken. The Internet Archive does not delete or "nuke" the data that is blocked by a robots.txt. Even though cough some people believe so (see parent thread).
Source: IA staffers.