rli
Re:Show HN: art fitting room
on Thursday December 20, 2012 @08:13AM (#4947521)
Nothing to be seen in Chrome v23.0.1271.95
Re:Ask HN: How are you dealing with scraping hits from EC2 machines?
on Wednesday December 12, 2012 @09:25AM (#4910142)
robots.txt can be ignored, it's just a reference for honest spiders. I think the way described above, of listing top requestors, doing statistics and then automating blocking is indeed the best way. Could also be there's a blocklist or two around of malicious scrapers. And if there isn't, that's a new business proposal.