Re:102TB of New Crawl Data Available
by malandrew on Wednesday November 27, 2013 @11:59PM
(#6813095)
I guess what I was suggesting is "given enough eyeballs, all spam and poor quality content is shallow"
There is definitely a benefit in using the community to identify valuable subsets and then individually putting your energy towards building discovery/search products around that subset.
Seriously - they give you an easy way to create these subsets yourself[1]. That is a much better solution than them trying to anticipate the exact needs of every potential client.
[1] http://commoncrawl.org/get-started/