I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.

https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results.

Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.

Thanks. I'll bring this to the attention of the bots and DDoS teams.

Yacy is 18 years old and not exactly obscure. If your bots team is unaware of it it's because they've chosen to be ignorant of it.

A search engine which is not run centrally by one organization on infrastructure in a known network, but rather something like YaCy where individual users run crawler nodes on networks that vary over time.

Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.

Yeah, I certainly don't want those crawlers anywhere near my servers. Block by default and allow site admins to unblock should they want to seems like the best way. It is also already how it works with Cloudflare.