News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by r_singh on Monday September 13, 2021 @03:33PM (#28515716)
Exactly!

While there are scraping APIs that unblock requests and charge for them, I'd love to learn more about how they work....

Re:How to Crawl the Web with Scrapy by Eikon on Monday September 13, 2021 @04:19PM (#28516314)
They use residential proxies with altered clients and / or headless browsers. Cloudflare's bot protection mostly makes use of TLS fingerprinting, and thus pretty easy to bypass.
Re:How to Crawl the Web with Scrapy by wswope on Monday September 13, 2021 @04:26PM (#28516417)
Scraping is a cat and mouse game that’ll vary a lot by site. I’m far from an expert and welcome correction here, but the two big tricks that’ll go a long way AFAIK are using a residential proxy service (never tried one - they tend to be quite shady), and using a webdriver-type setup like Selenium or Puppeteer to mock realistic behavior (though IIRC you have to obfuscate both those systems since they’re detectable via JS).