Re:How to Crawl the Web with Scrapy
by ramraj07 on Tuesday September 14, 2021 @07:51AM
(#28523288)
How do you deal with these god awful SPAs though, when you just do requests..
[ Parent
| on HN ]
Re:How to Crawl the Web with Scrapy
by TechBro8615 on Tuesday September 14, 2021 @11:45AM
(#28526518)
SPAs tend to be some of the easiest sites to scrape (at least if you’re building targeted scrapers), because they come with a production-ready API out of the box.
You can just watch the network tab of dev tools to know what endpoints to hit. Same applies to mobile apps – before you reach for BeautifulSoup, maybe check to see if the website has a mobile app too. It’s usually worth setting up a mitmproxy to see what API you might be able to scrape.
Re:How to Crawl the Web with Scrapy
by artembugara on Tuesday September 14, 2021 @07:52AM
(#28523302)
We do it as a part of post-processing, we'd have to make one more call.
Quite annoying though