In theory someone could do something similar with terms, or you could first use URLs to filter the text size you download into Elastic or Solr and do your own custom search that way.
The indexes are really neat though, I highly recommend playing with them.
So setting up a new search engine that way would require going to every site and convincing them to notify you of changes. Wouldn't that be even more limited than the current Cloudflare whitelist system? At least there's some chance you can get around the whitelist system.
I mean, I get it, then sites can send to any number of indexers, but let's be honest, like you say, any new search engine has to get sites to push data out to them. That's just not. going. to. happen.
Cloudflare's protection is to guard against traffic spikes and automated malicious attacks; Google and social sites are allow-listed because they're trustable entities with well-defined online presences and a vested interest (generally) in not breaking sites. "How do we do away with the need to put up an anti-automated-traffic screen plus whitelist" is in the same category of problem as "How do we change the modern web to address automated malicious attacks?"
This simply isn't true. By my reading, Cloudflare implies any automated traffic that doesn't openly advertise itself as such is bad. They certainly seem to be happy to assist you in blocking it regardless of whether it was actually malicious or actually causing traffic problems. (https://blog.cloudflare.com/super-bot-fight-mode/)
More reliable services for those willing to de-pssudonymize and declare themselves in an auditable fashion.
What if you want to make a simple link previewer? An abstract crawler for scientific articles? Most websites are behind cloudflare which will block/captcha you, but happily whitelist only google & major social sites. Tha answer is measures that bring the web back to basics, not this over-SEOed bot infested ecosystem. FANGS succeeded in sucking out all the information of the web, but they suck at creating protocols that are interoperable (even twitter now needs its own tags!).
Incidentally, maybe the next search engine should use a push-system , where websites will ping to it whenever they have updates. If the engine has unique features, it might be actually seen as a measure to reduce the loads from bots.