Re:How to scrape anything on the web and not get caught
by dewey on Sunday April 22, 2018 @02:41AM
(#16895002)
You could use https://oxylabs.io/ or buy some regular VPN accounts and build a http proxy wrapper around that. Extremely cheap and works well. There’s a lot of existing projects on Github for that too.
[ Parent
| on HN ]
Re:How to scrape anything on the web and not get caught
by ikeboy on Sunday April 22, 2018 @02:53AM
(#16895038)
oxylabs.io seems more expensive than luminati, minimum of $178/month. Not clear if they charge for bandwidth.
Problem with VPN is it's shared and hard to get a lot of IPs, any specific ones I could get say 100 dedicated US IPs for a reasonable price?
Re:How to scrape anything on the web and not get caught
by dewey on Sunday April 22, 2018 @04:24AM
(#16895259)
Depends on your scale and what you negotiate. The shared nature of VPN is usually not a huge problem but depends on your use case. Most VPN providers have a better deal for bigger customers so you can just buy multiple accounts in bulk with each having for example 5 connections and use that. For US specifically it often happens that they have 100s of servers and vpn configs so you can build something out of that.