Re:Robots.txt is a suicide note
by migrantgeek on Tuesday May 10, 2011 @03:20AM
(#2531699)
"Archiveteam projects are different because we run closely-monitored, highly targeted crawls."
Maybe, but does that make robots.txt useless? I don't think so.
Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing?
The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves?
If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.
Maybe, but does that make robots.txt useless? I don't think so.
Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing?
The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves?
If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.