I agree 100% (I come from a "use all the cores with threads and shared memory to get linear speedups on many workloads" world). But I think at this point the python leadership has committed to the "every time the GIL-removal looks plausible, find some more single-core speedups to stave off the change". Put another way: I've given up on the hope that python will ever multicore the way I think it should multicore.

I'd be tickled if Python would singlecore the way I think it should singlecore. Unfortunately, it remains considerably slower even than JavaScript (never mind Go, Java, etc). A lot of bad choices that can't easily be undone without breaking compatibility for a handful of packages, and the Python leadership seems unable or unwilling to guide the community through those changes.

I had a similar problem last month where I needed to compute some results on multiple datasets (with a little bit of concurrency per dataset). In the end I just ported the code to Ada which makes it very easy to create task hierarchies. You can define variables of task types on the 'stack' and the program leaves the enclosing block when the variables can go out of scope after every task has finished. I can now easily put all cores to maximum use. The list comprehensions in Python became a little bit more verbose though because I had to use an older revision (2012) instead of the new 2022 revision.

Have you checked out the Ray library? I use it for this type of scenario and it works well, without as much effort as it would take otherwise.

SharedMemory is a part of the multiprocessing module