I wouldn't do that. It's better to stay under the radar IMO. They usually won't notice the traffic if it's just for personal reasons. And who says the internet can't be used for automatic retrieval?

I'm not going to pound websites with requests every minute of course but I think this falls under legitimate use. Whether I click the button myself or just schedule a script to do it shouldn't matter so much.

But if you ask you draw attention to it and get their legal dept involved (after all most websites are not run by a single webmaster in their bedroom) who will most likely say no because legal people are hesitant to commit to anything.

But maybe my usecase is different. I just scrape stuff to check if something I want is back in stock, to download my daily pdf newspaper I pay for, archive forum posts I've written, stuff like that. I don't index whole sites.

But yeah I do make sure I don't bombard them with requests, though this is more from a "staying under the radar" point of view. And indeed to avoid triggering stuff like cloudflare.

But if you're scraping to run your own search engine and offer the results to the public the situation is much more complex of course, both technically and legally.

I've never actually reached out to a webmaster to ask permission, but I think that's a great idea. (They may even have some suggestions for a better way to achieve what I'm doing.)

How do you typically find contact info for someone like that?

I'm running very generous politeness intervals at the moment to try and ensure I'm not a nuisance - one query every two seconds.

If you can't find the address on the website, sometimes it's in robots.txt, you could also try support@domain, admin@domain or webmaster@domain.

Contact forms can as well, usually they seem to get forwarded to the IT department if they look like they have IT stuff in them, and I've had reasonable success getting hold of the right people that way.

I'll give that a shot!

Do you know if this is generally an in-house position for companies that use third party platforms?

I ask because Workday has been the absolute bane of my indexing existence, and I suspect they make it hard so they can own distribution of jobs to approved search engines. (Makes it easier to upcharge that way, I suppose.)

If the administrator for the job site is the Workday customer (i.e. Qualcomm or NXP or whoever is using Workday to host their job ads), I'd suspect I'd have a chance at getting a better way to index their jobs. (My god, I'd love API access if that's a thing I can get. I'd be a fly on the wall in most cases - one index a day is plenty for my purposes!)

If their security is any good, their firewall may tarpit you anyway if they can see you are spidering links quicker than a human can read, your browser agent offers no clues and/or your ip address (range) offers no clues.

This is a nice approach. I generally leave a project specific email in the request headers with a similar short summary of my goals.

Yeah, my User-Agent is "search.marginalia.nu", and my contact information is not hard to find on that site. Misbehaving bots are very annoying and it's incredibly frustrating when you can't get hold of the owner to tell them about it.