Re:Google's robots.txt
by anon4 on Sunday February 23, 2014 @04:04AM
(#7285358)
It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
[ Parent
| on HN ]
There are no comments on this comment yet.