Re:Fast scraping in Python with Asyncio
by bwood on Tuesday March 04, 2014 @01:24AM
(#7338820)
Clearly this would be faster than using the requests library to scrape pages in serial, but would it also be faster than using the requests library in several threads at once? Besides the simplified syntax, what is the advantage of asyncio over a threaded scraper?
[ Parent
| on HN ]
I would say the performance are equivalent, with maybe some overhead if there is some heavy thread spawning. I'll run some benchmarks at some point.
One of the biggest advantages is the fact this is not multi-threading, so context switches can only happen at precise points (yield from). This makes it easier to reason about the code.