News for hackers. Stuff that matters.
Re:How to scrape anything on the web and not get caught by sram1337 on Sunday April 22, 2018 @01:21AM (#16894829)
I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
Re:How to scrape anything on the web and not get caught by dingaling on Sunday April 22, 2018 @05:01AM (#16895342)
Can't you use a service like Panopticlick to 'tune' the browser fingerprint to a majority percentile? Particularly if running in clean VM images.
Re:How to scrape anything on the web and not get caught by JeanMarcS on Sunday April 22, 2018 @01:52AM (#16894895)
Rotating user agents ?
Re:How to scrape anything on the web and not get caught by dredmorbius on Sunday April 22, 2018 @04:07AM (#16895215)
Minimally effective, or outright detrimental, in my own limited testing.
Re:How to scrape anything on the web and not get caught by jaytaylor on Sunday April 22, 2018 @01:57AM (#16894910)
Some proxies will also include your IP address in the form of an "X-Forwarded-For" header.