Re:Google's robots.txt
by bmelton on Sunday February 23, 2014 @03:13AM
(#7285305)
That's amazing. Know of anything like it for Nginx?
[ Parent
| on HN ]
It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}'
Not tested. :)