Re:What one may find in robots.txt
by cmdrfred on Monday May 18, 2015 @03:19PM
(#9566316)
I'm with you on this, not for huge sites but if even 1% of sites used this behavior malicious bots would be forced to honor robots.txt
[ Parent
| on HN ]
There are no comments on this comment yet.