In my five years of python I've run up against this boundary at least once. In your list I would

* take out #2. if something can make use of multiple nodes it can usually make even better use of multi-core parallelization (which affects both computational and memory bandwidth performance). multi-node comes with a much higher communications overhead, so there's a relatively wide range of applications that scale well on multi-core but not multi-node.

* add that #3 comes up as soon as you have complex data structures to share. Serializing and Deserializing (by default with pickle) is a huge overhead for anything a bit more involved. If you design for this from the start you can be fine, but often these things grow and eat up bigger and bigger usecases until you run against the GIL. This basically happens with anything that has enough data and users and need - hey I heard your scheduler tool works well for the cafeteria, I'm sure it can handle our global operations right?

* about #4 - see the previous point.

Once (or a few times) in 5 years puts this problem into the "not worth(ROI) solving" bucket for me.

Those few times, put down the hammer and use some other tool for those not naillike jobs.

Here's the thing: Python, especially 3.6, is such a well rounded language that all other major limitations have IMO been solved already. In my view the GIL is the main one left, and reason to pause and think whether python is a good idea at the start of a project. Removing it is therefore worth it, and would also give a nice additional incentive for everyone to switch to python 3.x, so we don't have to keep on maintaining 2.7 with the same code (i.e. the worst of both worlds).

Certainly not worth it for one person to tackle the GIL, but a million people running into it a few times in 5 years, and I think it's economical.

#1 & #2: Consumer CPUs are now pushing 16 cores & 32 threads. Python is limited to ~1/20th of what a single box is capable of. That's a pretty big bottleneck.

#4: Even if you're just talking message passing sending a message between threads is in the 10s of nanoseconds while between processes is 10s of microseconds. That's a ~1000x slowdown on core communication. Given that CPU cores are not getting any faster, that's a pretty big hit to efficiency to take. Similarly simply moving data between processes is expensive, while moving data between threads is free.

Moving data between threads is only free to the extent that synchronization is free. Maybe you could say that moving immutable data between threads is free but I don't think you can say its free in general ... Doing so significantly undersells the complexity that comes with shared memory concurrency.

You seem to be conflating moving with sharing. Moving between threads is always free[1] regardless of if it's mutable or immutable, and there's no concurrency issues at all since it's a move.

Move means the sender no longer has a reference. As in, std::move, rust's ownership transfer, webworker's transferables, etc...

1: Yes there's a single synchronize point where the handoff happens, but this is part of sending a message at all. It's also independent to the size & complexity of the payload itself when we're talking multi-threaded instead of multi-process. You have that exact same sync point that costs the exact same regardless of whether your message consists of a single byte or a multi-gigabyte structure.

Ah I see -- Can you actually describe ownership in python sufficiently well to be able to describe this move operation for any useful python data structures?

Generically ownership is simply who has a reference to the object.

So if a.foo = b, then a 'owns' b. A 'move' is simply handing a different object the reference, then dropping your own reference. For example:

  a.foo = b // a 'owns' b
  c.foo = a.foo // a & c share b, however if the next line is:
  a.foo = None // a has 'moved' b to c, since c now has the only reference to b.

Some languages have codified this to make the contract part of the language, but it doesn't need any first-class language support. It's just a pattern at the end of the day.

You are right about the majority of what you said, but I am pedantically picking on one point. CPU cores are getting faster, but they aren't doing it with clock speed, they are dispatching more instructions per cycle or otherwise making the work faster.

IPC gains per generation are vanishingly tiny, if they exist at all. Skylake -> Kaby Lake, for example, had no IPC improvements at all. A very small clock bump to the various tiers was it.

Even if you look over a large generation gap there's only a ~20% IPC improvement going from an i7-2600K to an i7-7700K ( https://www.hardocp.com/article/2017/01/13/kaby_lake_7700k_v... )

6 years & a shrink from 32nm to 14nm and all it can muster is +20%. Cores are just not getting faster by any meaningful amount.

My compiles have gotten more than 20% faster so something is making my newere machines faster than my older machines.

That it is not 10x as fast I blame on AMD for not being as competitive as they could have been.

Yeah - I remember when five year old CPUs were basically useless!

(Kaby Lake is basically a new stepping of Skylake - if intel wasn't having problems with new process nodes it likely wouldn't have been released at all, and if it was it would've been used for a one-off chip in the same generation ala the 4770K)

The efficiency hit is very dependent on how large your computation chunks are. If the computation per message batch is on order of 100 ms, it would be <10% loss.

Assuming very small messages that are rarely sent then yes, the hit of multi process is not going to be your biggest issue.

Your criteria 2, 3, and 4 doesn't make much sense to me. We often have workloads that require multiple boxes, but we still want to make effective use of each box. Common server hardware has dozens of cores, which requires a lot of parallelism to fully utilize. The GIL hinders that, even when most of the work doesn't hold the GIL (see Amdahl's law)

Python multiprocessing doesn't work well with a lot of external libraries. For example, CUDA doesn't work across forks and many system resources can be shared across threads but not processes. Python objects must be pickled to be sent to another process, but not all objects can be pickled (including some built-in objects like tracebacks).

A lot of different parallel programming models can be built on top of threads (shared memory, fork-join, message passing), and to a certain extent they can be mixed. That's not true of Python multiprocessing, which only allows a narrow form of message passing. (It's also buggy, has internal race conditions, and easily leaks resources.)

The problem for CPython is that it may not be possible to remove the GIL without breaking the C API, and a lot of the benefit of Python is the huge number of high-quality packages, many of which use the C API.

CPython doesn't have any reservations about breaking the Python API between minor versions, so why care about the C API? I get where you're coming from, but they've already shown they don't care much for compatibility, so I don't see why that's a big obstacle.

Removing the GIL (in a non-braindead way) likely entails breaking all existing code using the C API. PyPy could do so without breaking cpyext, by maintaining the illusion of a GIL whenever control passes to cpyext.

That makes sense, I hadn't thought about the extent of the breakage.

Does it lock the GIL so numpy can release it again immediately afterwards?

Perhaps it makes the unlock call a no-op before numpy tries to unlock it.

> (see Amdahl's law)

Amdahl's law bears little relevance to throughput computing (i.e. most servers).

> (It's also buggy, has internal race conditions, and easily leaks resources.)

There is also at least one memory corruption bug in multiprocessing (linked a few months back by a fellow HN reader).

There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.

Can't this already be handled by calling out to a C/C++ or FORTRAN procedure that processes the data in multiple threads? For number crunching, Python is almost exclusively used as glue.

You CAN handle it, but why should you have to? If it's possible to remove that barrier, then it absolutely should be removed. If the only answer to a problem is "use another language", then the language in question has a limitation that needs to be addressed.

It is not a limitation at all in this case. Python is just a front to Tensorflow and similar libraries/frameworks so GIL doesn't matter there.

Today's machine learning and data science students don't know how to code in those languages. They know python, and maybe java.

Don't forget R... shudders

What's wrong with R? I know it's not a programmers language but it's great for getting things done.

So work on data that can not be broken down into smaller chunks? That makes sense, and is something I never come across.

I'm sure they can be broken down into smaller chunks, but is it more efficient if they aren't broken down and instead shared memory is used? If you want parallelism you're obviously already worried about performance.

Something like the web worker primitives might work there (transferables & sharing read only data).

Are those applications often bottlenecked by the CPU, as opposed to GPU or data transfer?

The world of algorithms that run well on a CPU is still much, much bigger than the world of algorithms that run well on a GPU, even in machine learning.

And even if you're fortunate enough that Nvidia designs their GPUs to solve your problem, why should the CPU cores sit idle?

Motivation for removing the GIL is basically that when people hear about it they go "hmmm that doesn't sound good". Obviously many applications have been written in GIL languages and there aren't really many practical problems that can't be overcome easily.

I think it may be some Stockholm Syndrome -- people have worked very hard to get around the GIL, and they've come to expect its limitations and respect those solutions.

But I've never heard of someone asking for a GIL to be added to the JVM.

This! Try to implement a controlled task scheduler using multiprocessing and sooner or later you are going to hit some unexpected behavior, like - multiprocessing.Queue belly-upping for no reason, UNIX signals propagating throughout the process chain and killing them left and right, hitting some data/object which are not serializable etc. Getting multiprocessing to work right takes a LOT of careful efforts, which breaks the whole promise python.

I've since moved to clojure, which is a language designed with concurrency from ground up. Look at clojure's `atom` - it's basically what every beginning programmer expects from globally shared variables, minus the gymnastics of handling race conditions on your own.

Also, `core.async` is such a beautiful thing to work with for writing schedulers. Compared to this, python's asyncio is an unfunny joke.

I don't think python's GIL can be removed with ad-hoc locking. Nothing sort of complete re-implementation will do.

Exactly! If a feature is good it's worth adding, and the GIL in any language is not good.

People are just as quick to bemoan a language for not having something (generics, templates, pre-processors) because they see some perceived need, but a GIL is never one of those things.

+1 on this - what is more important for me is some kind of Numba LLVM jit to automatically optimize hotspots : kind of like the JVM hotspot compiler.

Numba already does some of this.

Additionally, I cannot help but wonder if the answer to these problems has been the JVM all along. Especially with JVM 9 and the Truffle framework - https://github.com/securesystemslab/zippy

I was just about to mention Graal & Truffle when I saw your post! I wasn't aware of ZipPy but it looks promising! Java 9 will provide a proper interface for Graal through JVMCI and is only 37 days away from GA [1]. With Graal supposedly only months away from GA [2], ZipPy may very well prove to be the future of high performance Python.

[1] http://www.java9countdown.xyz/ [2] https://www.infoq.com/presentations/polyglot-jvm-graal (see roughly 42:00 - 47:00)

EDIT: Wording.

Say you're running CPU-bound workers that need to load significant data into RAM - say, a machine learning or NLP model. The most cost-effective theoretical approach would be to have that in shared memory, so you're not paying for that RAM multiple times in order to fully utilize all cores. Even if you need multiple boxes, the cost savings per core would be substantial. My understanding is that multiprocessing makes you jump through hoops to set up that shared memory; this would make it largely transparent to the user while remaining performant. I haven't used multiprocessing in production, though, so I could be wildly off base there.

Unless your model actually consists of a large number of Python objects (and not a handful of PyObjects referencing something like a np array), there isn't really anything blocking you from doing so. You can have a master process map the blob of static data into a block of shared memory that's mapped by the secondary processes; ctypeslib lets you access it as a numpy array again.

Using multiprocessing is a pain to use, and it's slow.

If you’re looking for simple threaded multiprocessing, it’s not that hard/painful:

    from multiprocessing.dummy import Pool
    
    pool = Pool(num_threads)
    result = pool.map(your_func, your_objects)
    pool.close()
    pool.join()

Improve and/or complicate things from there.

This is a nice pattern and there are surprisingly many problems that can be solved that way. AFAIK you do not have to join() here as the processes die after the map call.

Often the challenge is a big amount of (hopefully read-only) data that you want to access in every 'your_func'. The naive solution is to copy the data, but this might blow your memory.

in what was is it a pain that threading is not?

With threading, all of your threads can refer to the same objects. Multiprocessing means you have multiple interpreters running. That means no shared memory, and communication over pretty slow queues. I've definitely wanted to have multithreaded Python programs where all threads referred to the same large read-only data structure. But I can't do this because of the GIL. I mean, I can, but it's pointless. I can't do this with multiprocessing because of the limitations on shared memory with multiprocessing.

Edit: I realize I'm contradicting myself here. No shared memory is a first approximation. You can have shared memory with multiprocessing, but most objects can't be shared.

And yet, if you could have what you want, would it actually be faster?

The costs of synchronizing mutable data between cores is surprisingly high. Any time your CPU thinks that the data that it has in its cache might not be what some other CPU has in its cache, the two have to coordinate what they are doing. And thanks to the fact that Python uses reference counting, data is constantly being changed even though you don't think that you're changing it.

Furthermore if you throw out the GIL for fine-grained locking, you then open up a world of potential problems such as deadlocks. Which look like "my program mysteriously froze". Life just got a lot more complicated.

It is easy to look at all of those cores and say, "I just want my program to use all of them!" But doing that and actually GETTING better performance is a lot trickier than it might seem.

Right, but like I said, I'd be fine with a read-only shared data structure. I have a problem that has a hefty data model. The problem can be decomposed and attacked in parallel, but the decomposition doesn't cut across the data. Right now I run n instances on n cores, but that means making n copies of a large data structure. This requires a lot of system memory, ruins any chance I have of not wrecking the cache (not that I have high hopes there, but still), and forces me into certain patterns, like using long-lived processes because it's expensive to set up the model, that I'd prefer to avoid.

You might want to look at https://stackoverflow.com/questions/17785275/share-large-rea... for inspiration.

If you need to share a large readonly structure, the best way IMO is that approach. Implement the structure in a low-level language that supports mmap (be very sure to make the whole structure be in the mmap'd block - it is easy to wind up with pointers to random other memory and you don't want that!) and have high performance accessors to use in your code.

Thanks for the link! Might be worth going down that path.

Good luck. Another benefit of this strategy is that you optimize that data structure using techniques that aren't available in higher languages. So, for instance, small trees can be set up to have all of the nodes of the tree very close together, improving the odds of a cache hit. You can switch from lots of small strings to having integers that index a lookup table of strings for display only.

The amount of work to do this is insane. Expect 10x what it took to write it in a high level language. But the performance can often be made 10-100x as well. Which is a giant payoff.

Thanks! I've already partly rewritten it in C once, but I misunderstood the access pattern and I ended up having a lot of cache misses. The speedup was measurable, but disappointing, and the prospect of doing another rewrite had put me off. I hadn't put two and two together about this being an effective way to share memory under multiprocessing until reading this thread, so it's worth revisiting now.

Yeah, sharing memory between processes is a very delicate ballet to perform. That said, sharing a read-only piece of data is way simpler than you'd expect, depending on size and your forking chain. The documentation could do a better job of explaining the nuances and provide more examples.

Care to elaborate? All I've seen in the docs is how to share arrays or C structures between processes. It would take a substantial rewrite to use either. Is there some kind of CoW mechanism I'm missing?

Serializing data for IPC is often undesirable (copies kill) which leads to multi process shared memory. Sharing memory across process boundaries safely is a problem you avoid entirely with threading. You still need to lock your data (or use immutable data), but the machinery is built into your implementation (and hopefully trustworthy).

It's been a while, and my memory is fuzzy, but I recall either pyodbc or pysybase reacting very poorly with the multiprocessing module. With multiprocessing, Python would segfault after fork. With threading, it would "work" albeit slowly. Also, IIRC, it did not matter if the module was imported before or after the fork, still segfaulted. I never had the time to try and track down the issue that was causing it, though, deadlines and all that.

You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker. Also, copying large datasets between processes is not efficient. And also, there are cases where the fan-out approach is not the best way of parallelizing a task, and passing information back up to a parent task is more complicated than necessary.

"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker."

the multiprocessing library allows you to launch multiple processes using your function definitions. It's almost the same as the multithreading library but does not share data.

It seems the real problem, as you pointed out, is the additional memory. I didn't consider situations where each process would need an identical large data set, instead of just a small chunk to work on.

It gets more interesting when you have a large data set that's required for the computation, but as you compute, you may discover partial solutions that can be cached and used by other workers.

So not only a large read-only data set, but also a read-write cache used by all workers. This sort of thing is relatively easy with threads, but basically impossible with multiprocessing.

Depending on where you want to go and the application, such things may be good idea for a low number of workers but can become a major bottleneck.

To add to what everyone else said, if you need transactional semantics, its much simpler in multiple threads. With multiple processes (local or remote), you can't simply share an atomic data structure or a lock, you have to use a distributed lock or consensus algorithm, which are more complex and usually quite "chatty". If memory or network bandwidth are constrained, it may be especially desirable to eliminate this, but even if not, fast locking/transactions may be desirable regardless.

If you're using multiple processes for CPU-bound performance, why not squeeze as much as you can out of each CPU?

Just like you can share memory between processes, you can also share OS-level locks and semaphores between them. A distributed lock manager is not required for the single-node case.

I haven't written shared memory code in literally years, I just use Redis now.

It's just a low hanging fruit for perfs from the dev point of view. It's nice and useful, just nowhere as needed as most people asking for it pretend it is.