Say more about this. I'm not a software engineer by training, so I don't really know what "persist your state" would look like in this case.

I work on web scrapers at [day job] and can say the latter approach is far better, but the former is far more common. An implementation of the former could be as simple as "dump the current URL queue to a CSV file every minute".

As for doing this "properly" in a way that works at scale, my preferred way is with a producer-consumer pipeline and out-of-process shared queues in between each pair. So, for example, you have 4 queues: URL queue, verify queue, reponse queue, item queue and 5 stages: fetch (reading from URLQ and writing to verifyQ), response check (reading from verifyQ, writing good responses to reponseQ and bad response URLs back to fetchQ), parse (reading from responseQ and writing to itemQ) and publish (reading from itemQ and writing to database or whatever).

This is is be both horrendously overcomplicated and beautifully simple. I've implemented this both with straight up bash scripts and with dedicated queues and auto-scaling armies of docker containers.

One of the things I've been doing with this scraping project is "health checking" job posting links. There's nothing more annoying than clicking an interesting looking link on a job site, only to find it's been filled. (This is one of the lousiest parts of Indeed and similar, IMO.) I wrote some pretty simple routines

Caching solves the problem of potentially missing data while the scraper is running, but it doesn't really alleviate the network strain of requesting pages to see that they are actually still posted, valid job links.