Re:How to scrape anything on the web and not get caught
by sram1337 on Sunday April 22, 2018 @01:21AM
(#16894829)
I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
[ Parent
| on HN ]
Re:How to scrape anything on the web and not get caught
by dingaling on Sunday April 22, 2018 @05:01AM
(#16895342)
Can't you use a service like Panopticlick to 'tune' the browser fingerprint to a majority percentile? Particularly if running in clean VM images.
Re:How to scrape anything on the web and not get caught
by JeanMarcS on Sunday April 22, 2018 @01:52AM
(#16894895)
Rotating user agents ?
Re:How to scrape anything on the web and not get caught
by dredmorbius on Sunday April 22, 2018 @04:07AM
(#16895215)
Minimally effective, or outright detrimental, in my own limited testing.