Avoiding Webscraping Throttling Using Python and Tor as a Proxy

Re:Avoiding Webscraping Throttling Using Python and Tor as a... by steventhedev on Sunday July 14, 2019 @01:21AM (#20431801)

Sounds like you need to make things easier rather than harder. They have a clear economic incentive to scrape your site and will continue to do so. Rather than spending your employees time on it, open it up as an API. If someone starts hitting it too hard, send them an email to back off.

If your only competitive advantage is your pricing, then you're in a losing game. Otherwise, you're probably better off spending your efforts on making your product/service better.

[ Parent | on HN ]

Re:Avoiding Webscraping Throttling Using Python and Tor as a... by viraptor on Sunday July 14, 2019 @08:44AM (#20433078)

This approach is not very realistic. 1. API design and auth takes engineering time and potentially brings more traffic you need to pay for. 2. You don't have an email address to send "back off" to. Even if you had, no enforcement means no change. 3. Blocking / throttling takes much less time and money than creating a new endpoint.

[ Parent | on HN ]

Re:Avoiding Webscraping Throttling Using Python and Tor as a... by ForHackernews on Sunday July 14, 2019 @05:24AM (#20432551)

Charge your competitors for API tokens?

[ Parent | on HN ]