News for hackers. Stuff that matters.

A comment on How would you build an internet scale web crawler?. Parent comment

Re:How would you build an internet scale web crawler? by titanix2 on Friday April 06, 2018 @09:59AM (#16773240)
As for open-source crawler goes, I've heard of SpiderLing (http://corpus.tools/wiki/SpiderLing) which is somewhat specialized for building text corpus.
[ Parent | on HN ]

There are no comments on this comment yet.