Isn't multiprocessing almost syntactically equivalent to--but much more efficient than--threading in python?

The problem is the GIL : https://wiki.python.org/moin/GlobalInterpreterLock

Multiprocessing allows you to use all cores, but you can't share memory like in threads. So you have to Serialize/Deserialize a lot for exchange data.

If your serializations cost are low and need computation power, you should use multiprocessing. If you are just waiting I/O you can use threads.

BTW, modern python "ProcessPoolExecutor" is not designed for compute heavy batch like "Pool" from "multiprocessing".

Syntactically equivalent? The syntax is the same - both multiprocessing and threading are achieved through libraries and VM primitives, not by additional syntax.

But if you meant semantically equivalent, no they aren't - threading is shared memory and multiprocessing is message passing. If you were to draw a tree of approaches to concurrency and parallelism, you would probably make this division the root node bisecting the domain into two - it's the two fundamental approaches to doing concurrency.

> threading is shared memory and multiprocessing is message passing

In most cases, yes. You can use shared memory buffers for IPC, though, although it typically requires more effort.

It's an implementation difference with big practical implications. 'Fundamental divide at the root of all concurrency' seems more than a little overstated.

Almost every approach to concurrency fits into either shared memory or message passing.

Those that cross between the two are usually notable for doing that and describe themselves as a hybrid approach.

I think it divides the literature pretty cleanly in two.

Interesting, but mostly pointless IMHO.

Python can't do proper multithreading because of the GIL (excluding Jython and IronPython, of course), hence using threads in Python doesn't benefit anyone.

If you want to learn how to do multithreaded programming, pick a language where multithreading actually works - Java would be a good example. Or just take a look at a POSIX threads tutorial.

If you want to handle concurrency in Python, ignore multithreading. Take a look at the multiprocessing module, or, for a different paradigm, at asyncio or Twisted.

If your threads are doing I/O rather than computation, then the GIL is less of an issue because it will be released until an I/O operation completes. It's only for threads which contend for the CPU that the GIL is a problem.

Well this isn't the full story. If you have an I/O bound work-load, say you want to parallelize HTTP requests, threads work just fine.

If you have an I/O bound workload, a reactor or proactor approach are way better. No dealing with locks, race conditions, etc.

I never said it's wrong, it's just pointless. In Python, there's always a better alternative to multithreading which has no shortcomings.

That's assuming all the IO you can block on HAS support for non-threaded concurrency. Things that wrap C libraries may support threads, but not the reactor-of-the-day (twisted? tornado? etc.)

Nice video series. Why don't you work in a Linux environment?

Should it make a difference then?

I just wondered. Everyone recommends linux to me. I asked to get the idea. Not a criticism.

- historically unix/linux is nicer to process data as you want (linux shells + grep / sed / awk .. ), windows only recently got a workable shell.

- most if not all oss languages run on linux

- git used to be way faster for linux (may have changed since)

- programmers tend to enjoy system they can own, less the case on windows unless you work against the system (made for non tech saavy users)