> but at the same time, CloudFlare makes it basically impossible for any new search engine other than Google to spider the site

I hadn't heard about this, can anyone supply a link for more context?

Is there anything a Cloudflare customer can do to "opt in" to being scraped by other search engine bots?