Web Scraping 101 with Python

Re:Web Scraping 101 with Python by almost on Wednesday February 10, 2021 @11:45AM (#26091501)

I've found that a lot of the time that's not needed. Often you'll find the data as a JSON blob in the page and can just read it directly from there. Or find that there's an API endpoint that the javascript reads.

[ Parent | on HN ]

Re:Web Scraping 101 with Python by monkeybutton on Wednesday February 10, 2021 @12:17PM (#26091971)

I find this method works best. Skip looking at the page and instead watch all the network requests as the page loads.

[ Parent | on HN ]

Re:Web Scraping 101 with Python by WrtCdEvrydy on Wednesday February 10, 2021 @01:35PM (#26093039)

ZAP HUD Proxy is the best option here...

Load the page on it and it shows you all the request being made along with the payload as it happens.

Find the one you need, copy the data, endpoint and HTTP verb and recreate it in your language of choice :D

[ Parent | on HN ]