Re:Free 5 Billion Page Web Index Now Available from Common C...
by ahadrana on Tuesday November 08, 2011 @12:56PM
(#3211868)
We hear you. Could you define some criteria as to the type and size of sample data you would like to see? We are working on producing more targeted/limited collections, like perhaps all most recently published blog posts etc.
[ Parent
| on HN ]
Re:Free 5 Billion Page Web Index Now Available from Common C...
by showerst on Tuesday November 08, 2011 @01:59PM
(#3212210)
Perhaps two sets, one that's just a few hundred kilobytes that contains a few sample .arc files to test against the format, and then one larger 'training' set that's small enough to test against offline (maybe like 100MB?) but large enough to contain a good sample of the possible content.
Re:Free 5 Billion Page Web Index Now Available from Common C...
by dcnstrct on Wednesday November 09, 2011 @03:09AM
(#3214744)
Concur with this comment -- it might also help the community provide feedback on structure and ways to segment that data so that there are more directed efforts to consume small parts of the crawl for processing