News for hackers. Stuff that matters.
Re:Web Scraping in 2016 by monsoon22 on Tuesday August 23, 2016 @03:24PM (#12346693)
How do you circumvent cloud provider IP blocks? For example, one site blocks all requests from AWS EC2 servers.
Re:Web Scraping in 2016 by fake-name on Tuesday August 23, 2016 @03:28PM (#12346735)
None of the sites I'm scraping do that, mostly.

I'm not scraping high value sites like that (I mostly target amateur original content). It's not really of interest to other businesses. As such, I tend to just run into things like normal cloud-flare wrapped sites, and one place that tried to detect bots and return intentionally garbled data.

If I run into that sort of thing, I guess we'll see.

Re:Web Scraping in 2016 by nickysielicki on Wednesday August 24, 2016 @05:14PM (#12355304)
I've never used this, and it's incredibly shady considering the users probably do not realize that their Hola browser plugin does this, but Hola runs a paid VPN service where you can get thousands of low-bandwidth connections on unique residential IP addresses, provided generously through their "free" VPN users.... It's essentially a legitimate attempt at running a botnet as a service.

But if the end justifies the means... http://luminati.io/