Thanks, I certainly don't want to worry too much but, at the same time, the data is a big component of the value proposition in this case and likely what cost us more (more than the actual development of the service, for example).

So much so that we are likely to make more money by giving access to the data via API to other companies than by allowing "normal" people to search through the data and get paid a referral fee every time they click on something.

By why should we avoid the latter as an additional revenue stream? I'd rather try and make it more difficult for people to scrape it (and I don't want to ask people to accept T&C before making a search)

Thanks for the reply Paul

impose rate limits on IP, class C IP's, detect non-Javascript bots excluding search engines, block most major web hosts/cloud hosts (very doubtful a legit user is from Amazon Cloud IP).

As someone who bypassed paying fees for APIs by just scraping, I think you do have a legit concern. But don't be concerned about regular users scraping it, be concerned about potential buyers scraping it. Regular users scraping it can't do much damage.

thanks a lot, all good suggestions

What do you like to use for your scraping? I used python/BeautifulSoup back in the day but there may be better options now. Did you spoof other browser headers or did people really not care?