Re:Ask HN: How are you dealing with scraping hits from EC2 m...
by adrianoconnor on Wednesday December 12, 2012 @09:13AM
(#4910076)
Do you have a robots.txt? That's the standard way.
[ Parent
| on HN ]
Re:Ask HN: How are you dealing with scraping hits from EC2 m...
by dumbfounder on Wednesday December 12, 2012 @09:24AM
(#4910133)
Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.
Re:Ask HN: How are you dealing with scraping hits from EC2 m...
by ig1 on Wednesday December 12, 2012 @09:43AM
(#4910232)
Not really. "Scraping" just refers to extracting data from a site using an automated method, it doesn't have any connotations about the motivation or acceptability of the process.
Re:Ask HN: How are you dealing with scraping hits from EC2 m...
by rli on Wednesday December 12, 2012 @09:25AM
(#4910142)
robots.txt can be ignored, it's just a reference for honest spiders. I think the way described above, of listing top requestors, doing statistics and then automating blocking is indeed the best way. Could also be there's a blocklist or two around of malicious scrapers. And if there isn't, that's a new business proposal.
Re:Ask HN: How are you dealing with scraping hits from EC2 m...
by lifeguard on Wednesday December 12, 2012 @09:28AM
(#4910160)
That is the way to block spiders that obey the standards and enough do not that robots.txt is not a solution.