Re:How to Crawl the Web with Scrapy
by wswope on Monday September 13, 2021 @04:26PM
(#28516417)
Scraping is a cat and mouse game that’ll vary a lot by site. I’m far from an expert and welcome correction here, but the two big tricks that’ll go a long way AFAIK are using a residential proxy service (never tried one - they tend to be quite shady), and using a webdriver-type setup like Selenium or Puppeteer to mock realistic behavior (though IIRC you have to obfuscate both those systems since they’re detectable via JS).
[ Parent
| on HN ]
There are no comments on this comment yet.