Re:What one may find in robots.txt
by protomyth on Monday May 18, 2015 @11:07AM
(#9564861)
This is silly. If you have a user of your site going through your bots file and specifically going to directories listed as Disallow then you deal with that user. Blocking based on the robots.txt for a directory that doesn't exist anywhere but that file is fine. I did a two bad directory ban, it seemed to work fine.
[ Parent
| on HN ]
There are no comments on this comment yet.