Build a crawler to crawl million pages with only one machine in just 2 hours

Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @10:37PM (#13760854)

Thanks for your comment. If you have the same test with cloud server or some public website , perhaps it will decrease some.

I've used multiprocessing/threads/geven/asyncio before. And I will have a full test with these libraries.

Thanks again!

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by zepolen on Wednesday March 01, 2017 @10:21PM (#13771147)

As I said, the benchmark is flawed since it's dependent on the network pipe. It would be a good idea to run tests locally so you get a real maximum.

There are lots of factors involved which can completely skew benchmarks, for example, if you were scraping an average 10kb response instead of 'hello world' you would automatically be limited to 100req/s on a 10mbit pipe.

[ Parent | on HN ]