News for hackers. Stuff that matters.
Re:Python Asyncio: The Complete Guide by number6 on Thursday November 10, 2022 @12:06PM (#33549633)
Asyncio is not ment for CPU Bound Task but for IO Bound Tasks. You should have used multiprocessing. The problem you describe is exactly what asyncio is used for: saturate your bandwidth better.

maybe the right aproach would have been a Threadpool? Plus you don't have to refractor the task. Just let it run sync, but you can make it also async

Re:Python Asyncio: The Complete Guide by matsemann on Thursday November 10, 2022 @01:23PM (#33550851)
In other languages the async paradigm work for multiple kind of workflows, not just heavily IO bound ones.
Re:Python Asyncio: The Complete Guide by number6 on Thursday November 10, 2022 @01:32PM (#33550986)
Can you give me an example which languages do this?
Re:Python Asyncio: The Complete Guide by matsemann on Thursday November 10, 2022 @01:40PM (#33551117)
In Kotlin for instance you can run your coroutines with different "dispatchers", which have different behaviors. You can for instance use a dispatcher with multiple threads in a threadpool, run it in the main thread, a specific thread, in threads with low/high priority etc. Basically allowing you to write code using the async/coroutine paradigm, but then control its execution from the outside.
Re:Python Asyncio: The Complete Guide by tmh88j on Thursday November 10, 2022 @02:30PM (#33551788)
> In Kotlin for instance you can execute your coroutines with different "dispatchers", which have different behaviors. You can for instance use a dispatcher with multiple threads in a threadpool, run it in the main thread, a specific thread, in threads with low/high priority etc. Basically allowing you to write code using the async/coroutine paradigm, but then control its execution from the outside.

I'm not very experienced with Kotlin, but that sounds similar to python's run_in_executor [1]

[1] https://docs.python.org/3/library/asyncio-eventloop.html#asy...

Re:Python Asyncio: The Complete Guide by throwaway81523 on Monday November 14, 2022 @07:27PM (#33603507)
Look at seastar.io which is an async concurrency library for C++. It is painful in that it resembles node.js but it is very performant. I'd be interested in seeing a version adapted to use C++20 coroutines.
Re:Python Asyncio: The Complete Guide by qwertox on Thursday November 10, 2022 @04:39PM (#33553787)
But it only does so because you can move out that extra work from the async thread into a thread pool.

In Python there's no benefit in doing this due to the GIL, unless you're using a module which implements its own multithreading (for example in C). Python is not alone with this issue.

In other languages you're basically stepping out of the async paradigm in order to use threading in parallel. You can wait for result of the thread in the async loop without blocking.

I really enjoy using asyncio in Python for things where I have to do a lot of stuff in parallel, like executing remote scripts in a dozen of servers in parallel via AsyncSSH or for low workload servers which query databases.

In any case, what keeps me hooked on Python like a junkie is the `reload(module)` which I invoke via an inotify hook every time a file/module changes:

server.py

server_handler.py

reloader.py

server.py loads reloader.py (which sets up inotify) and that one takes care of reloading server_handler.py whenever it changes. server.py defers all the request/response handling to server_handler.py which can be edited on the fly so that the next request executes the new code. Instant hot reloading.

This also works very nice with asyncio (like aiohttp) and one ends up with very readable code.

If you need performance, then use Java, Rust, Go or C/C++, but for prototyping and tooling I absolutely love this approach with Python.

Re:Python Asyncio: The Complete Guide by dekhn on Thursday November 10, 2022 @01:38PM (#33551072)
any concurrency system that is properly designed supports both types of tasks.
Re:Python Asyncio: The Complete Guide by number6 on Thursday November 10, 2022 @01:41PM (#33551123)
I only know C#: https://learn.microsoft.com/en-us/dotnet/csharp/async#recogn...

There you also have to act differently based the task

Re:Python Asyncio: The Complete Guide by dekhn on Thursday November 10, 2022 @01:59PM (#33551368)
The docs on that page explicitly say you just have to provide a different parameter (basically, to force it to use a thread-like instead of select-like approach).

The docs on that page also point you to a different task library which does what I'd expect:

"If the work is appropriate for concurrency and parallelism, also consider using the Task Parallel Library."

I checked those docs and that is a framework that makes sense.

Re:Python Asyncio: The Complete Guide by phs2501 on Thursday November 10, 2022 @02:20PM (#33551639)
Not sure how that's different than Python asyncio's "run blocking stuff with `run_in_executor`, the result (future) of which integrates into the async event loop?
Re:Python Asyncio: The Complete Guide by dekhn on Thursday November 10, 2022 @04:10PM (#33553418)
it's threaded with the option to use 1:M, N:N or N:M. It's a mental model that includes both asyncio and threads, while async will always only be for IO-blocking calls that can be multiplexed.
Re:Python Asyncio: The Complete Guide by JosephRedfern on Thursday November 10, 2022 @02:12PM (#33551521)
Depending on the specifics, a threadpool proabably wouldn't help with a CPU bound task due to GIL limitations. `multiprocess.Pool` might (though process pools aren' without their overheads)
Re:Python Asyncio: The Complete Guide by jerf on Thursday November 10, 2022 @03:40PM (#33552981)
"Asyncio is not ment for CPU Bound Task but for IO Bound Tasks."

You can amplify this; pure Python is not meant for CPU-bound tasks. If you have a pure Python task, you've CPU optimized it somewhat, and it's still too slow, the answer is, get out of pure Python.

There's half-a-dozen options that still leave you essentially in Python land (just not pure Python anymore) like cython or NumPy, and of course dozens of other languages that leave Python entirely.

To accelerate a pure Python task to the speed that you can get out of a single thread of a more efficient language, you have to perfectly parallelize your task across upwards of 40 CPUs (conservatively!), because that's how slow pure Python is. asyncio isn't a solution to this, but neither is any sort of multiprocessing of the pure Python either.

This is not criticism. This is just engineering reality. Pure Python is a fun language, but a very slow one.