Would there be a way to crowdsource the web crawling effort? Crawling@Home? I think that even with this you'd need a legal department to deal with takedown requests and illegal material.