I'm looking for an approach to writing concurrent code with parallelism that is elegant and easy to understand and hard to introduce bugs. This requires alternative programming approaches and in my perspective, alternative notations.
One such design uses monotonic state machines which can only move in one direction. I've designed a syntax and written a parser and very toy runtime for the notation.
https://github.com/samsquire/ideas5#56-stateful-circle-progr...
https://github.com/samsquire/ideas4#558-state-machine-formul...
The idea is inspired by LMAX Disruptor and queuing systems.
That's basically what all machine learning code written in Python does. It calls out to libraries that can themselves parallelize, use the GPU, etc. And then gets the answer back. You get parallelism without any special Python support.
It's 2023 already. 1988 called.
It turns out that when the programmer doesn't understand what the tool says, managers believe the programmer and throw out the tool. Coverity was finding itself in situations where they were finding real bugs, and being punished for it by losing the sale. So they removed the checks for those bugs.
I'll revisit my opinion of multithreaded code when things like that stop happening. In the meantime there are models of how to run code on multiple threads that work well enough with different primitives. See Erlang, Go, and Rust for three of them. Also, if you squint sideways, microservices. (Though most people set up microservices in a way that makes debugging problematic. Topic for another day.)
Source?
> As an example, for many years we gave up on checkers that flagged concurrency errors; while finding such errors was not too difficult, explaining them to many users was.
(And thanks; I was also wondering about that)
https://en.wikipedia.org/wiki/Never_Gonna_Give_You_Up
On 12 March 1988, "Never Gonna Give You Up" reached number one in the American Billboard Hot 100 chart after having been played by resident DJ, Larry Levan, at the Paradise Garage in 1987. The single topped the charts in 25 countries worldwide.
Now we have languages and language facilities (consider Rust, Haskell, and others) to make it much safer. Same with green threads and what Go and now Java does.
Out of curiosity, have you done any Rust programming and used Rayon?
It's hard to convey how easy and impactful multi-threading can be if properly enclosed in a safe abstraction.
It's hard to convey how difficult it would be to retrofit python to be able to truly "enclose multithreading in a safe abstraction".
See https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.h... for more.
Also, no matter how much you wish it otherwise, retrofitting concurrency on an existing project guarantees that you'll wind up with subtle concurrency bugs. You might not encounter them often, and they're hard to spot, but they'll be there.
Furthermore existing libraries that expect to be single-threaded are now a potential source of concurrency bugs. And there is no particular reason to expect the authors of said libraries to have either the skills or interest to track those bugs down and fix them. Nor do I expect that multi-threaded enthusiasts who use those libraries in unwise ways will recognize the potential problems. Particularly not in a dynamic language like Python that doesn't have great tool support for tracking down such bugs in an automated way.
As a result if "no GIL" ever gets merged, I expect that the whole Python ecosystem will get much worse as well. But that's no skin off of my back - I've learned plenty of languages. I can simply learn one that hasn't (yet) made this category of mistake.