Build a crawler to crawl million pages with only one machine in just 2 hours

Re:Build a crawler to crawl million pages with only one mach... by zepolen on Tuesday February 28, 2017 @01:00PM (#13755890)

Could also use multiprocessing, got about ~500req/s returning a 'hello world' response (which the article also does). The article does about 300req/s but that's because he saturates his pipe. The reality is the article might be faster than 1,000,000/hour.

    from multiprocessing import Pool
    from requests import get
    urls = 1000 * ['http://localhost/hello']
    def scrape(url):
        return get(url).text
    p = Pool(40)
    results = p.map(scrape, urls)

~2.2 seconds on a dual core 2.2ghz

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @10:37PM (#13760854)

Thanks for your comment. If you have the same test with cloud server or some public website , perhaps it will decrease some.

I've used multiprocessing/threads/geven/asyncio before. And I will have a full test with these libraries.

Thanks again!

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by zepolen on Wednesday March 01, 2017 @10:21PM (#13771147)

As I said, the benchmark is flawed since it's dependent on the network pipe. It would be a good idea to run tests locally so you get a real maximum.

There are lots of factors involved which can completely skew benchmarks, for example, if you were scraping an average 10kb response instead of 'hello world' you would automatically be limited to 100req/s on a 10mbit pipe.

[ Parent | on HN ]