News for hackers. Stuff that matters.

A comment on How to crawl a quarter billion webpages in 40 hours (2012). Parent comment

Re:How to crawl a quarter billion webpages in 40 hours (2012) by _trampeltier on Thursday July 05, 2018 @01:03PM (#17464867)
I did run a Yacy web crawler (P2P websearch https://yacy.net) a while ago. As far I remember I just saw Yandex for a few times disallowed in the robots.txt when I had trouble with crawling a site. Mostly I just got an empty website for my Yacy crawler instead the "real" Website.
[ Parent | on HN ]

There are no comments on this comment yet.