Re:Ask HN: how do comparison websites prevent scraping?
by AznHisoka on Monday January 16, 2012 @01:22PM
(#3471774)
impose rate limits on IP, class C IP's, detect non-Javascript bots excluding search engines, block most major web hosts/cloud hosts (very doubtful a legit user is from Amazon Cloud IP).
As someone who bypassed paying fees for APIs by just scraping, I think you do have a legit concern. But don't be concerned about regular users scraping it, be concerned about potential buyers scraping it. Regular users scraping it can't do much damage.
As someone who bypassed paying fees for APIs by just scraping, I think you do have a legit concern. But don't be concerned about regular users scraping it, be concerned about potential buyers scraping it. Regular users scraping it can't do much damage.