I've built native support for RSS and JSON feeds for discovery on high-quality sites, and I'm also tracking estimated content refresh rate to know when to re-fetch.
(I'm at a point where I've given myself too many signals to flag against, and finding the right tuning while starting to build an index, where I'm also actively evolving the schema and figuring out how to prioritize content I return in SERPs, is a problem. My current thinking is just a series of differentiated set of rules that populate the crawl queue for different reasons—seeking appropriate breadth—but then I'll definitely hit the re-crawl issue, particularly around sites I'm scoring as high quality.)
There's actually a paradox with new content that should make it less interesting to crawl, which is that the odds content will vanish or change is inversely proportional to its age. If something has been around for 10 years, it's a fairly safe bet it will be tomorrow. If something has been around for 10 hours, it's a coin-toss.
Aggressively seeking out fresh content is probably a waste. If anything, fresh content should be regarded with suspicion. Maybe probe it a few days later to make sure it's still there before adding it to the index.
I'm working on a federated search tool. I've built a few different sectors (I've called them "realms") that I care about (programming, automotive, racing, fediverse) and found that I need pretty different heuristics depending. Example: car forums circa ~2005 are a treasure trove of valuable information, but 17 year old posts about programming are (in general) less interesting. That informs a lot about how URLs need or do not need to be re-crawled.
I do think offering different filters like that is probably a good option to having the search engine try to mind-read what you want based on spying on your historical queries and machine learning haruspicy.
Right now a lot of this work is manual. Kinda not long-term viable, but for now it's not too bad. Kick off a script every now and again and come back a week later and see how it went.
After that, there needs to be some sort of prioritization. Maybe a job that randomly probes URLs and flags dead/changed links for priority re-crawling. I think using RSS feeds to detect new links is also a tool that might be useful.