News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours (2012) by dredmorbius on Thursday July 05, 2018 @12:17PM (#17464433)
Right: that's "crawl, but share".

Both models are fairly isomorphic, modulo the crawl / search locus. The key is on agreed data structures, practices, and access.

There are no comments on this comment yet.