News for hackers. Stuff that matters.
Re:Screenshots Forever and Ever Until You Can’t Stand it by StavrosK on Wednesday August 20, 2014 @05:32AM (#8201660)
But why retroactively remove the data? The original owner was fine with holding it, why should the snapshot be deleted because a completely different person wants his completely different website to not be crawled?
Re:Screenshots Forever and Ever Until You Can’t Stand it by bjt on Wednesday August 20, 2014 @12:50PM (#8203128)
It's hard for a bot to understand concepts of 'owner' and 'completely different person' based on the data they have available. Companies can use this robots.txt feature to un-index old marketing content after a re-branding, for example. Or after an acquisition.
Re:Screenshots Forever and Ever Until You Can’t Stand it by StavrosK on Wednesday August 20, 2014 @12:55PM (#8203159)
Sure, but, surely, the bot has timestamps saying "robots.txt allowed me to keep these documents last time I spidered them". Why do they have to be retroactively removed? robots.txt only disallows spidering, it doesn't mandate that you should delete all the data you've already spidered.
Re:Screenshots Forever and Ever Until You Can’t Stand it by db48x on Thursday August 21, 2014 @01:31AM (#8206030)
Because most of the problems come from people who want to hide old material that they didn't realize was being indexed. The automatic behavior is simple and easy to implement, and doesn't require any human intervention.