Re:Ask HN: how do comparison websites prevent scraping?
by AznHisoka on Monday January 16, 2012 @01:22PM
(#3471774)
impose rate limits on IP, class C IP's, detect non-Javascript bots excluding search engines, block most major web hosts/cloud hosts (very doubtful a legit user is from Amazon Cloud IP).
As someone who bypassed paying fees for APIs by just scraping, I think you do have a legit concern. But don't be concerned about regular users scraping it, be concerned about potential buyers scraping it. Regular users scraping it can't do much damage.
So much so that we are likely to make more money by giving access to the data via API to other companies than by allowing "normal" people to search through the data and get paid a referral fee every time they click on something.
By why should we avoid the latter as an additional revenue stream? I'd rather try and make it more difficult for people to scrape it (and I don't want to ask people to accept T&C before making a search)
Thanks for the reply Paul