> most communities are non-technical and probably won't be able to set up their own crawler

They can use a solution which already integrates the search. Forums and CMSes are a good target for that. Then you can say "I'd like my search to look at widgetlovers.com too" - and you get their sitemap + featured external links, because they run FooPress that supports it.

Kind of the same as sitemap we already produce for Google.