I dont believe python will ever have atomic operations, even if it had - they still incur significant overhead for concurrency control.
sharing state between threads is such a narow niche use case, this pattern is practically solved by memcached/redis for larger scale python based systems
Yes, but there are plenty of problems where that is more development overhead, but less runtime overhead than IPC.
at which point you lose benefit of no-GIL, because you now have to introduce redis and kafka into the system
Why?
Yeah, if you are building a system that is, say, serving web requests, and have an internet scale potential market, success might mean that.
Not every system works that way. A simulation system with defined parameters doesn’t grow in scale if it becomes more popular, you just have more people running isolated instances that don’t depend on each other. Plenty of other applications scale that way rather than the “SaaS that serves ever more clients” way.
Still lots of complexity and awkwardness that can be avoided if you can do threading instead of processes. Like Promotheus scraping from a non-webserver python app is a pita, as you need a new process and lots of communication, vs just plug and play as in other languages.
Or just the insane resource usage. Had a java app serving multiple orders of magnitude more customers running on a few containers. Our current python app needs multiple deployments with different entry points, and about 15x the amount of containers.
That policy is over.
As the last years have shown, no alternative implementation can get off the ground due to C extensions and compatibility concerns, and CPython is now relied on for many large applications. It no longer makes sense to prioritise a simple implementation over performance.
Finely-grained locks are useful. Even when you end up scaling between machines, it can be useful to have many threads in one memory space to maximize what you get out of one machine.
We're moving up to hundreds of cores; Python often being stuck only being able to use a couple while tightly coupling state has been unfortunate.
Building applications that run on multiple machines is at least one order of magnitude more complex and thus slower (in development velocity), so needlessly building an application to work distributedly is just bad engineering.
And "one beefy machine" has a very high limit, so by the time you actually outgrow it you usually have tons of resources available to help rewrite things.
You might think that you can simply spin M processes per machine instead but now you have N*M servers instead of N servers that are M times faster. In many cases this means you have significantly higher overheads: slower startups, a lot more RAM usage, more network IO etc.
Outside of a few embarrassingly parallel problem, two-level parallelism is usually the highest performance approach.
It is the norm. „Kafka scale“ problems are not the norm.
Outside of python, no it really isn't, it's the norm.
And even within python itself: That redis connection IS shared state.
I am thinking about multithreading every day to try make it easier to use. I journal about it in my ideas journals.
Anything that benefits from both parallelism and replacing IPC overhead with shared data between parallel tasks.