Re:What one may find in robots.txt
by gpvos on Monday May 18, 2015 @06:58AM
(#9563690)
Curiosity?
[ Parent
| on HN ]
This is silly. If you have a user of your site going through your bots file and specifically going to directories listed as Disallow then you deal with that user. Blocking based on the robots.txt for a directory that doesn't exist anywhere but that file is fine. I did a two bad directory ban, it seemed to work fine.
Curiosity killed the cat, or so they say.
As long as the block lasts not more than a few days, as he says in a nearby comment, I don't think it's much of a problem.
Oh, believe me they only last a few days at most. I don't use an infinite IP blocklist. It has like 100 IP's or so. When one goes in, one must come out. And lets just say there are enough badly behaved bots around that it doesn't take much time for 100 IP's to rotate.
Re:What one may find in robots.txt
by beaumartinez on Monday May 18, 2015 @07:16AM
(#9563752)
"But satisfaction brought him back."