How to crawl a quarter billion webpages in 40 hours (2012)

Re:How to crawl a quarter billion webpages in 40 hours (2012) by TomK32 on Thursday July 05, 2018 @08:59AM (#17462881)

I don't see any issue at all if the bot is respecting the robots.txt file. Any malicious user will figure some other evil way anyways, be it Nutch or a network of intelligent lightbulbs.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours (2012) by merb on Thursday July 05, 2018 @09:54AM (#17463248)

so you will actually respect:

    User-agent: google
    Allow: /
    
    User-agent: *
    Disallow: /

well, yes it's good behavior to actually respect it, but well I've seen such robots.txt already which makes it really painful to create a competing search engine.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours (2012) by taeric on Thursday July 05, 2018 @12:32PM (#17464593)

This has me ridiculously curious now. Is that common? Other than a random sampling of sites I go to, there a good way to get numbers on how often this is used?

Edit: In my scanning, I have to confess that wikipedia's robot file is the best. Fairly heavily commented on why the rules are there. https://en.wikipedia.org/robots.txt

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours (2012) by benfrederickson on Thursday July 05, 2018 @01:45PM (#17465226)

I analyzed the top 1 million robots.txt files looking for sites that allow google and block everyone else here: https://www.benfrederickson.com/robots-txt-analysis/ - it's a relatively common pattern for major websites

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours (2012) by _trampeltier on Thursday July 05, 2018 @01:03PM (#17464867)

I did run a Yacy web crawler (P2P websearch https://yacy.net) a while ago. As far I remember I just saw Yandex for a few times disallowed in the robots.txt when I had trouble with crawling a site. Mostly I just got an empty website for my Yacy crawler instead the "real" Website.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours (2012) by gcb0 on Thursday July 05, 2018 @03:03PM (#17465861)

just do like browsers did with user agent strings. call your bot "botx (google crawler compatible)" and crawl everything that allows Google bot without any weight on your conscience.

[ Parent | on HN ]