News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by ramraj07 on Tuesday September 14, 2021 @07:51AM (#28523288)
How do you deal with these god awful SPAs though, when you just do requests..
Re:How to Crawl the Web with Scrapy by TechBro8615 on Tuesday September 14, 2021 @11:45AM (#28526518)
SPAs tend to be some of the easiest sites to scrape (at least if you’re building targeted scrapers), because they come with a production-ready API out of the box.

You can just watch the network tab of dev tools to know what endpoints to hit. Same applies to mobile apps – before you reach for BeautifulSoup, maybe check to see if the website has a mobile app too. It’s usually worth setting up a mitmproxy to see what API you might be able to scrape.

Re:How to Crawl the Web with Scrapy by artembugara on Tuesday September 14, 2021 @07:52AM (#28523302)
We do it as a part of post-processing, we'd have to make one more call.

Quite annoying though