News for hackers. Stuff that matters.
Re:Notes on Writing Web Scrapers by cushychicken on Thursday December 02, 2021 @02:48PM (#29421472)
I'll give that a shot!

Do you know if this is generally an in-house position for companies that use third party platforms?

I ask because Workday has been the absolute bane of my indexing existence, and I suspect they make it hard so they can own distribution of jobs to approved search engines. (Makes it easier to upcharge that way, I suppose.)

If the administrator for the job site is the Workday customer (i.e. Qualcomm or NXP or whoever is using Workday to host their job ads), I'd suspect I'd have a chance at getting a better way to index their jobs. (My god, I'd love API access if that's a thing I can get. I'd be a fly on the wall in most cases - one index a day is plenty for my purposes!)

Re:Notes on Writing Web Scrapers by Terry_Roll on Thursday December 02, 2021 @06:06PM (#29424161)
If their security is any good, their firewall may tarpit you anyway if they can see you are spidering links quicker than a human can read, your browser agent offers no clues and/or your ip address (range) offers no clues.