Re:Build a crawler to crawl million pages with only one mach...
by plantpark on Tuesday February 28, 2017 @10:17PM
(#13760720)
Thanks for your comment.
You are right, asyncio and aiohttp is a great combination. I've used them both before. Though aiohttp cost less memory than requests, aiohttp does't support https proxy. This is the only one that isn't so perfect.
Besides , asyncio is something the same as concurrency of celery. I will have a test which one will have a better performance.
You are right, asyncio and aiohttp is a great combination. I've used them both before. Though aiohttp cost less memory than requests, aiohttp does't support https proxy. This is the only one that isn't so perfect.
Besides , asyncio is something the same as concurrency of celery. I will have a test which one will have a better performance.
Thanks for your comment again.