News for hackers. Stuff that matters.
Re:Nuitka: a Python compiler written in Python by amelius on Thursday June 17, 2021 @10:31AM (#27539972)
Not really. You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. Unless of course you translate all of your data to C too, at which point you are really doing everything in C (data and code).
Re:Nuitka: a Python compiler written in Python by quietbritishjim on Thursday June 17, 2021 @12:15PM (#27541287)
OK, let's put it like this: In some contexts, but not all, Python supports a useful amount of parallelism.

I have definitely worked on Python projects in the past that were easily parallisable and worked absoluately fine. I simply created threads with threading.Thread, passed messages around with queue.Queue and used some standard Python modules to process those messages. The messages were independent enough from each other that I didn't have any issues with marshalling access to them, while being large enough that the overhead of mulithreading was still outweighed by its benefits. Rewriting the whole thing in C++ (etc.) would have made negligable difference to performance.

I can certainly imagine a program similar to what you describe, where there are lots of messages that operate on a single monolythic data structure, which couldn't easily be made to work with true parallelisation in Python.

Re:Nuitka: a Python compiler written in Python by incrudible on Thursday June 17, 2021 @10:59AM (#27540272)
> You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads.

You probably do not want to do that in the first place. The reason that the GIL still exists is that the overhead of fine-grained locking for Python datastructures would negate the speedup from multithreading.

Re:Nuitka: a Python compiler written in Python by amelius on Thursday June 17, 2021 @11:41AM (#27540836)
Which is why Python doesn't really support parallelism.

Other language (like GoLang) do support parallelism but they have a more advanced runtime and don't use a GIL.

Re:Nuitka: a Python compiler written in Python by incrudible on Thursday June 17, 2021 @02:42PM (#27542976)
It's not specific to Python. If you want to do fine-grained "parallelism" on a set of complex shared datastructures, your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. That's not necessarily going to result in a speedup over a lock-free single-threaded implementation (that you didn't implement to compare against).

In the case of Python, the single-threaded implementation was already there, the implementation that removed the GIL in favor of fine-grained locking failed to deliver the goods.

If you want to actually take advantage of parallelism, you want data structures that are amenable to it, which is basically sections of arrays without a lot of potential read/write conflicts over multiple threads. If you're at that point, you probably want to use C/C++/Rust anyway, you don't want speed up something that's already 100x slower (the interpreter) by parallelizing it. Python offers that with C-extensions like NumPy.

Re:Nuitka: a Python compiler written in Python by amelius on Friday June 18, 2021 @02:33AM (#27547905)
> your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff.

Not if you're reading most of the time and are using a modern concurrent GC.

Re:Nuitka: a Python compiler written in Python by incrudible on Friday June 18, 2021 @08:03AM (#27549470)
It doesn't matter if you're reading or writing, if there's a potential conflict you need to lock, or you need some complex and probably buggy "lock-free" data structure. The complexity and memory overhead of a concurrent GC just goes on top of that.
Re:Nuitka: a Python compiler written in Python by quietbritishjim on Sunday June 20, 2021 @01:32PM (#27572165)
Locking an uncontended lock is cheap (at least compared to a contended one) so actual conflict does matter more than potential conflict.