Re:A Raspberry Pi as a decent residential proxy
by city41 on Monday June 08, 2020 @09:20PM
(#23462643)
Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
Re:A Raspberry Pi as a decent residential proxy
by AlexITC on Monday June 08, 2020 @09:24PM
(#23462666)
Actually, that's what I'm doing.
In the post I detailed that I run a service (https://cazadescuentos.net) which is a browser extension to look for discounts.
Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go.
So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
Re:A Raspberry Pi as a decent residential proxy
by city41 on Monday June 08, 2020 @09:52PM
(#23462844)
Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.