Ask HN: Best way to block an EC2-hosted scraper?

Re:Ask HN: Best way to block an EC2-hosted scraper? by eli on Sunday January 09, 2011 @01:25PM (#2085977)

I assume you're running Apache? Have you taken a look at ModSecurity?

I found it a little tricky to set up (the core rules it ships with by default are way too aggressive, IMHO), but it does exactly what you're asking: block requests by regex.

[ Parent | on HN ]

Re:Ask HN: Best way to block an EC2-hosted scraper? by brandnewlow on Sunday January 09, 2011 @01:54PM (#2086057)

We're on a hybrid nginx/apache setup.

I just added some lines to my nginx.conf though that appear to be working based on tips from here and elsewhere:

location ~* /neighborhoods/[-\w]+/feed/?$ { deny all; }

[ Parent | on HN ]