News for hackers. Stuff that matters.

A comment on Build a crawler to crawl million pages with only one machine in just 2 hours. Parent comment

Re:Build a crawler to crawl million pages with only one mach... by rckclmbr on Tuesday February 28, 2017 @01:00PM (#13755885)
Check out scrapy, written in python. It also uses asynchronous io. Also, asyncio in python3 should make writing an async crawler a lot easier nowdays.
[ Parent | on HN ]
Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @10:12PM (#13760672)
Thanks for your comment. This article is a tutorial that tell people how to build a distributed system with docker and celry easily.

Scrapy is good framework for crawler. I used it before, but it doesn't have some features I want. Writing a brand new one is easier for me.

Of course , I am open to compare the special performances of these two crawlers. Welcome to have a discuss more technical details here.

Thanks for your comment again.

[ Parent | on HN ]