I intend to at least support other search engines by adding !bangs for them and recommending them in the UI if you didn't find the results you're looking for. I don't think that crawling is something that is easily distributed across independent orgs, though.
e.g opencrawl, internet-archive, archiveteam
It strikes me the resources to crawl, update, and manage/index data is a common problem.