I could see it being useful for gathering data via an API and returning it in an array, like if an API doesn't return an array of JSON objects but just an object, this could be useful for that use case. In general as long as it's not processor bound like computational tasks but may not be synchronous this could be useful. It also naturally extends the async for syntax to list comprehension.

It would be really nice if somehow these kinds of tasks could be tied into the multiprocessing package, so that async could handle processor intensive applications well.

This. This is essentially what you can do with the AsParallel() LINQ method, or iterating over an IEnumerable of tasks generated by async methods does in C#.

What about 'as_completed' in concurrent.futures? From my understanding, they fixed the performance issues with ProcessPoolExecutor in 3.4.

I recently tried to parallelize some non trivial code I have with multiprocessing.map/imap/imap_unordered - the state of parallel computing in python is really poor have to say. If the program's CPU runtime has gotten so long that you'd think about parallelizing, its datastructures tend to be too much to push through the pickle bottleneck and no gain would be had. Except of course number crunching stuff for which there's numpy. Bottom line is, you can do parallel computing, but trying to parallelize after the fact will usually fail, you have to plan for it from the very beginning due to no shared memory, which would add significant time until your first version is up and running

I long for a dynamic but strictly typed language with rich standard lib that gets this right. Nim?

You should consider Clojure with its optional typing library. I used Clojure, and later converted a single-threaded application into multi-threaded with dramatic gains. I did not need to plan in advance for parallelization and did not do so. Despite that, the conversion was fairly pain free. The application did the following:

1. read csv 2. create data structure 3. calculate statistics for each column (max values, unique values, etc.) 4. combine those statistics with original columns to generate new columns 5. write new columns to new csv

Processing each type of statistics in its own thread, combined with promises, allowed the processors to independently work on the same data structure. Because of the need to handle non-numeric values in numeric columns, e.g. convert nulls to zeroes, matrix manipulations would not have been appropriate.

I just wanted to point out a type of situation from my experience that goes against your generalization, to give a more complete picture.

Thanks, I'll consider clojure in the future. How would you say its standard lib and third party libraries hold up against python?

With my point about the need to plan ahead I specifically meant python by the way, and I think all runtimes that can't make use of concurrent multithreading have this in common. As soon as you can do multithreading with shared memory you become a lot more flexible in parallelising later on, and you can adjust your code style accordingly to accommodate this without much overhead. That's why the GIL IMO is so unfortunate, it's a real shame that Python 3 wasn't able to get rid of this when it started with a clean slate.

I see now what you meant. I apologize for my misunderstanding in that to be a general statement. Knowing that, I agree with your statements much more now. From what I can tell, the multi-process architecture in Python would require everything you mentioned. Not removing the GIL is indeed unfortunate.

Clojure is niche in a way. It is a Lisp dialect. However, it is based on the JVM and has built-in conveniences for using Java functions, classes, and libraries. Therefore, comparing libraries for enterprise applications is more accurately Clojure+Java vs. Python. My expertise is not in either language ecosystem, but my impression is that Python has better libraries for data analysis and as a system scripting language, and Clojure+Java is better for general applications and everything else.

One benefit of Clojure that makes parallelization easy is the immutable data structures. New data structures are cheaply made by seamlessly referencing originating data structures. Example: making a copy of array A and appending an item as array B, means that array B is mostly a reference to array A. Immutable data structures also offer expectations of behavior that mutable state naturally offends, such as avoiding race conditions or locks. In the case of my earlier example, I simply added some "future" wrapper functions and "dereference" annotations [1]. The entire program would execute to reach the final output, then work in reverse order to trigger and plan the future functions. Order of execution was based on dependencies, not strictly on the order of the code. It was fascinating was to watch the logs for when functions would start or complete. If you are familiar with operations planning and critical path analysis, the overall run-time reflected the critical path.

[1]: https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/c...

Its a bit immature at the moment, but here's a shout out for Julia. If you are used to python, numpy you will not feel out of place.

BTW, as far as I recall Nim is statically typed, please correct me if I am wrong.

Yes, Nim is static, however I've been told using lots of type inference and unions it can get almost the ease of use of dynamic languages. I never did something with it so far though, so I can't say whether that really holds up. My main concern would still be libraries - hard to beat python or the JVM in that regard I think.

Although the standard lib is still improving, Nim is doing it right.

async / await for on anything that implement the interface, not just asyncio. Which means it works for Future tied to multiprocessing pools, so yes, you can use async / await for you CPU bound program.

even if you assume it's not useful (i'm sure it is; arrays of API requests you have to process for example), it's probably a laudable goal and worth the effort to make things standard. i was surprised when i saw this and thought "oh you can't await in a comprehension?"; i expected you to be able to.