Build a crawler to crawl million pages with only one machine in just 2 hours

Re:Build a crawler to crawl million pages with only one mach... by wopwopwop on Tuesday February 28, 2017 @12:41PM (#13755631)

Because crawlers spend most of their time waiting, you might as well use python's gevents.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by greglindahl on Tuesday February 28, 2017 @02:16PM (#13756844)

Python 3's asyncio and aiohttp do the job for me -- I can crawl several times as fast as the article's 111 qps with just a single process. https://github.com/cocrawler/cocrawler

Of course, it matters what you're doing with the page content, and how you're managing your metadata, and all that.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @10:17PM (#13760720)

Thanks for your comment.

You are right, asyncio and aiohttp is a great combination. I've used them both before. Though aiohttp cost less memory than requests, aiohttp does't support https proxy. This is the only one that isn't so perfect.

Besides , asyncio is something the same as concurrency of celery. I will have a test which one will have a better performance.

Thanks for your comment again.

[ Parent | on HN ]