News for hackers. Stuff that matters.

A comment on Show HN: Bringing multithreading to Python's async event loop.

Re:Show HN: Bringing multithreading to Python's async event ... by noident on Saturday September 28, 2024 @04:27PM (#41682746)
> It got me wondering if it was actually possible to make Python’s async event loop work with multiple threads.

There is built-in support for this. Take a look at loop.run_in_executor. You can await something scheduled in a separate Thread/ProcessPoolExecutor.

Granted, this is different than making the async library end-to-end multi-threaded as you seem to be trying to do, but it does seem worth mentioning in this context. You _can_ have async and multiple threads at the same time!

[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by nbsande on Sunday September 29, 2024 @02:20AM (#41685302)
run_in_executor is pretty powerful for running sync code in async code, but my use case was more making async code utlize the cpu better. I think, just using run_in_executor would add a lot of complication and changes to how you use async await. But great point none the less!
[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by zo1 on Thursday October 03, 2024 @04:45AM (#41728628)
I've had plenty of success using run_in_executor in production. You basically just use the decorator, and it mostly just works.
[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by sandeep1998 on Thursday October 03, 2024 @01:07PM (#41732640)
Which decorator?
[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by zo1 on Thursday October 03, 2024 @02:21PM (#41733366)
I went back to look at some of the old code as my memory was hazy. It was tornado's implementation of the run_on_executor method that we used, which is used as a decorator.

https://www.tornadoweb.org/en/stable/concurrent.html#tornado...

[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by noident on Thursday October 03, 2024 @12:48PM (#41732451)
> but my use case was more making async code utlize the cpu better

But run_in_executor achieves that as well! If you use no-GIL Python and a thread pool (or GIL Python with a Process pool), you will utilize more CPU cores.

[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by d0mine on Thursday October 03, 2024 @05:39PM (#41735320)
run_in_executor that can be spelled as asyncio.to_thread() won't help utilizing cpu for pure Python code. It may help with blocking I/O, or cpu-heavy C extensions that release GIL. Otherwise, you have to utilize different processes to take full advantage of cpus (there are also subinterpreters (same process, separate GILs) but that exotic).
[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by noident on Sunday October 06, 2024 @03:24PM (#41759506)
>run_in_executor that can be spelled as asyncio.to_thread() won't help utilizing cpu for pure Python code

That isn't true. If you use a ProcessPoolExecutor as the target instead of the default executor, you will use multiple processes in pure Python code.

[ Parent | on HN ]
Re:Show HN: Bringing multithreading to Python's async event ... by d0mine on Monday October 07, 2024 @03:15PM (#41769848)
the default executor must be ThreadPoolExecutor. But you are right you can pass ProcessPoolExecutor too.

https://docs.python.org/3/library/asyncio-eventloop.html#asy...

[ Parent | on HN ]