News for hackers. Stuff that matters.

A comment on Ask HN: Best way to block an EC2-hosted scraper?.

Re:Ask HN: Best way to block an EC2-hosted scraper? by bretpiatt on Sunday January 09, 2011 @01:56PM (#2086068)
You can use varnish to cache all of these requests and just never expire them. Make the page as small as possible. It'll make the overhead of serving the requests as low as possible.

http://www.varnish-cache.org/

[ Parent | on HN ]
Re:Ask HN: Best way to block an EC2-hosted scraper? by buro9 on Sunday January 09, 2011 @02:18PM (#2086121)
You can also use Varnish ACLs to block the EC2 IP addresses.

And if they have a really obvious user-agent you could block that using Varnish: http://omninoggin.com/web-development/block-unwanted-spam-bo...

And if you're not afraid of using Varnish's VCL InlineC stuff, then you could add rate limiting to Varnish: http://drcarter.info/2010/04/how-fighting-against-scraping-u...

Basically... Varnish has become my Swiss army tool for rejecting crap traffic (and caching good traffic obviously).

[ Parent | on HN ]