I do not believe, as the archiveteam.org seem to think I do, that I have either stopped bots interfering with CGI scripts or spidering semi-private, temporary content. Indeed, my server logs show clearly enough that bots do these things all the time. Instead, the bot has violated my site's policy and its IP address will be considered for an iptables ban. It is a matter of which bots I consider fair game for rough treatment.
This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.
Quite. My robots.txt reads:
I do not believe, as the archiveteam.org seem to think I do, that I have either stopped bots interfering with CGI scripts or spidering semi-private, temporary content. Indeed, my server logs show clearly enough that bots do these things all the time. Instead, the bot has violated my site's policy and its IP address will be considered for an iptables ban. It is a matter of which bots I consider fair game for rough treatment.This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.