Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Ask HN: Best way to block an EC2-hosted scraper?. Parent comment
And if they have a really obvious user-agent you could block that using Varnish: http://omninoggin.com/web-development/block-unwanted-spam-bo...
And if you're not afraid of using Varnish's VCL InlineC stuff, then you could add rate limiting to Varnish: http://drcarter.info/2010/04/how-fighting-against-scraping-u...
Basically... Varnish has become my Swiss army tool for rejecting crap traffic (and caching good traffic obviously).
There are no comments on this comment yet.
And if they have a really obvious user-agent you could block that using Varnish: http://omninoggin.com/web-development/block-unwanted-spam-bo...
And if you're not afraid of using Varnish's VCL InlineC stuff, then you could add rate limiting to Varnish: http://drcarter.info/2010/04/how-fighting-against-scraping-u...
Basically... Varnish has become my Swiss army tool for rejecting crap traffic (and caching good traffic obviously).