Build a crawler to crawl million pages with only one machine in just 2 hours

Re:Build a crawler to crawl million pages with only one mach... by beejiu on Tuesday February 28, 2017 @12:42PM (#13755640)

I've just run a small test (crawling a server running locally) and it comes out at 243 pages per second with one process. This crawls a webpage, adds its links to the queue and saves the URL in a Redis set. This is running on a Macbook Pro.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by dchuk on Tuesday February 28, 2017 @03:29PM (#13757540)

So you eliminated the biggest cause of slow down in crawling, network latency, and are asserting yours is faster?

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by beejiu on Tuesday February 28, 2017 @03:57PM (#13757744)

The selling point of Node.js is asynchronous I/O. I'm sure you mean bandwidth rather than network latency - in which case that is really not a limiting factor when running in a datacenter (40 Gbps in at Linode for example).

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @09:53PM (#13760561)

Some library in python ,such as asyncio or gevent could do some work asynchronously and efficently. I will have a test later for these library. In the meanwhile , welcome to post more details about asynchronous of Node.js. Thanks for your comment again!

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by gbrits on Tuesday February 28, 2017 @04:01PM (#13757780)

Network latency would be a valid concern for avg. time/req. not for throughput (necessarily)

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by j_s on Tuesday February 28, 2017 @01:42PM (#13756433)

Nice! (compared to the article's 100,000 pages / 15 minutes / 60 seconds = ~110 pages per second)

Did you happen to track memory usage at all? It would take a while to settle down, for sure.

I'm always interested in the amount of overhead Docker brings to the table. No biggie either way, thanks for sharing these details.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @09:48PM (#13760533)

Thanks for your comment, I will have a test with a local server later and find out what's the upper limit of mine

[ Parent | on HN ]