News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by tamaharbor on Monday September 13, 2021 @05:56PM (#28517394)
Any suggestions regarding how to scrape Java-based websites? (For example, harness racing entries and results from: https://racing.ustrotting.com/default.aspx).
Re:How to Crawl the Web with Scrapy by mod on Monday September 13, 2021 @08:05PM (#28518503)
Generally you use devtools to find out what endpoints the javascript is requesting data from, then you make an identically-styled request.
Re:How to Crawl the Web with Scrapy by jcun4128 on Monday September 13, 2021 @06:39PM (#28517835)
What's wrong with it? That seems like a server-side rendered page/easier to deal with than waiting for JS to load.