Re:A Look Inside Our 210TB 2012 Web Corpus
by dsinha on Thursday August 15, 2013 @09:35PM
(#6221722)
Isn't there also the additional factor that webservers sometimes allow only the major search engines to crawl? If so, with something like this, should it gain popularity, and as more apps start using it, you'd hope more webservers allow the common crawler to crawl their websites which they might not if everyone were doing it individually...thinking aloud...
[ Parent
| on HN ]
There are no comments on this comment yet.