News for hackers. Stuff that matters.
Re:Notes on Writing Web Scrapers by marginalia_nu on Thursday December 02, 2021 @02:42PM (#29421391)
If you can't find the address on the website, sometimes it's in robots.txt, you could also try support@domain, admin@domain or webmaster@domain.

Contact forms can as well, usually they seem to get forwarded to the IT department if they look like they have IT stuff in them, and I've had reasonable success getting hold of the right people that way.

Re:Notes on Writing Web Scrapers by cushychicken on Thursday December 02, 2021 @02:48PM (#29421472)
I'll give that a shot!

Do you know if this is generally an in-house position for companies that use third party platforms?

I ask because Workday has been the absolute bane of my indexing existence, and I suspect they make it hard so they can own distribution of jobs to approved search engines. (Makes it easier to upcharge that way, I suppose.)

If the administrator for the job site is the Workday customer (i.e. Qualcomm or NXP or whoever is using Workday to host their job ads), I'd suspect I'd have a chance at getting a better way to index their jobs. (My god, I'd love API access if that's a thing I can get. I'd be a fly on the wall in most cases - one index a day is plenty for my purposes!)

Re:Notes on Writing Web Scrapers by Terry_Roll on Thursday December 02, 2021 @06:06PM (#29424161)
If their security is any good, their firewall may tarpit you anyway if they can see you are spidering links quicker than a human can read, your browser agent offers no clues and/or your ip address (range) offers no clues.