Re:Notes on Writing Web Scrapers
by cushychicken on Saturday December 04, 2021 @07:23AM
(#29440404)
Thanks for going down the Workday scraping rabbit hole with me. :)
Did you pull this from the browser console's "Copy as cURL" function?
I tried this with some success (there's even a utility for translating cURL to Python - imagine that! https://curlconverter.com/) but I had some issues after a while, probably because the cookie/session token expired.
Create a request mimicking this curl command:
curl 'https://nxp.wd3.myworkdayjobs.com/wday/cxs/nxp/careers/jobs' \ -H 'Connection: keep-alive' \ -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Accept-Language: en-US' \ -H 'sec-ch-ua-mobile: ?1' \ -H 'User-Agent: Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Mobile Safari/537.36' \ -H 'sec-ch-ua-platform: "Android"' \ -H 'Origin: https://nxp.wd3.myworkdayjobs.com' \ -H 'Sec-Fetch-Site: same-origin' \ -H 'Sec-Fetch-Mode: cors' \ -H 'Sec-Fetch-Dest: empty' \ -H 'Referer: https://nxp.wd3.myworkdayjobs.com/en-US/careers?p=4' \
Change the offset by +20 (second to last row) each time until you reach the desired number of jobs. May need some changes but that's the general approach!