Re:A Look Inside Our 210TB 2012 Web Corpus
by frederi on Tuesday August 13, 2013 @11:00PM
(#6209725)
Just because you can do it without much effort doesn't mean less experienced people can. Crawling can be a barrier to some people.
[ Parent
| on HN ]
To be honest a simple crawler is a very simple thing to write. If someone had issues getting that going I think they are going to have issues with the data volume anyway. LisaG answered why the 3 million data set though and I agree with the reasoning.