News for hackers. Stuff that matters.
Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @10:17PM (#13760720)
Thanks for your comment.

You are right, asyncio and aiohttp is a great combination. I've used them both before. Though aiohttp cost less memory than requests, aiohttp does't support https proxy. This is the only one that isn't so perfect.

Besides , asyncio is something the same as concurrency of celery. I will have a test which one will have a better performance.

Thanks for your comment again.

There are no comments on this comment yet.