Re:Asynchronous programming and cooperative multitasking
by greglindahl on Sunday February 10, 2019 @11:42AM
(#19128465)
Even in the python async/await world, you can get preemptable multitasking by adding in a few ProcessPoolExecutor threads. I've found this to be extremely effective in a web crawler: the main thread is 100% cooperative and does all of the network I/O on a single core, while cpu-hungry activities like webpage parsing are done by ProcessPoolExecutor processes on other cores. The code is pretty and I can fully use an entire server with what is conceptually a single crawler.
asyncio.create_subprocess_shell() is another way to get a preemptable thing in the Python async/await world.
asyncio.create_subprocess_shell() is another way to get a preemptable thing in the Python async/await world.