maltz
Re:Mastering Web Scraping in Python: Crawling from Scratch
on Thursday August 12, 2021 @02:16AM (#28152607)
You should probably take a look at this.
https://www.zenrows.com/blog/stealth-web-scraping-in-python-...
Re:Mastering Web Scraping in Python: Crawling from Scratch
on Thursday August 12, 2021 @02:10AM (#28152577)
YouTube has "var ytInitialData" & "var ytInitialPlayerResponse" params hardcoded in HTML.
No need to run JS!
Re:Mastering Web Scraping in Python: Crawling from Scratch
on Wednesday August 11, 2021 @05:43PM (#28148189)
Yep, seems so.
https://www.zenrows.com/pricing
Re:Mastering Web Scraping in Python: Crawling from Scratch
on Wednesday August 11, 2021 @01:03PM (#28144645)
If pages are constructed client-side, the content you are looking for is either hardcoded as JSON in the HTML or loaded via XHR request.
Scrape that.
Re:Mastering Web Scraping in Python: Crawling from Scratch
on Wednesday August 11, 2021 @12:15PM (#28143925)
Playwright.
It can be easily used with JS, Python, Go, Java, etc.
Re:Mastering Web Scraping in Python: Crawling from Scratch
on Wednesday August 11, 2021 @11:51AM (#28143578)
It's part of a series of blog posts that talks explicitly about crawling. There are indeed other links that do better explaining advanced extraction techniques.
Extraction => https://www.zenrows.com/blog/mastering-web-scraping-in-pytho...
Avoid blocking => https://www.zenrows.com/blog/stealth-web-scraping-in-python-...