Re:How to crawl a quarter billion webpages in 40 hours (2012)
by jdironman on Thursday July 05, 2018 @11:30AM
(#17464031)
Or possibly a common open resource from which services can pull regularly updated crawl data. I'm not sure if what I am thinking of exists or is feasible or not, just a thought from someone with little experience in crawling in general. Although, then that would put load on a single source and someone has to pay for those resources. Maybe if there was a distributed resource then, I wonder if block-chain technology or something similar factor into such a niche.
[ Parent
| on HN ]
Re:How to crawl a quarter billion webpages in 40 hours (2012)
by dredmorbius on Thursday July 05, 2018 @12:17PM
(#17464433)
Right: that's "crawl, but share".
Both models are fairly isomorphic, modulo the crawl / search locus. The key is on agreed data structures, practices, and access.
Re:How to crawl a quarter billion webpages in 40 hours (2012)
by mindcrime on Thursday July 05, 2018 @03:22PM
(#17466022)
That sounds a lot like what CommonCrawl does, and they are a good start. But from what I've seen, if you wanted to build a really competitive search engine, you'd still have to complement that with your own crawling. Not sure though, as I haven't done a deep dive into this topic.