News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by wswope on Monday September 13, 2021 @04:26PM (#28516417)
Scraping is a cat and mouse game that’ll vary a lot by site. I’m far from an expert and welcome correction here, but the two big tricks that’ll go a long way AFAIK are using a residential proxy service (never tried one - they tend to be quite shady), and using a webdriver-type setup like Selenium or Puppeteer to mock realistic behavior (though IIRC you have to obfuscate both those systems since they’re detectable via JS).

There are no comments on this comment yet.