News for hackers. Stuff that matters.

A comment on The Changing "Guarantees" Given by Python's Global Interpreter Lock.

Re:The Changing "Guarantees" Given by Python's Global Interp... by zzzeek on Friday November 17, 2023 @09:41AM (#38304240)
these examples of "what one would assume to be atomic" did not seem useful to me, they looked like things that are obviously not threadsafe.

a more interesting example is something like this:

   # setup
   l = []

   # thread A
   l.extend([1, 2, 3])


   # thread B
   l.extend([4, 5, 6])
is the resulting list always within the set of [1,2,3,4,5,6] or [4,5,6,1,2,3] ? or are the two sets of numbers randomly interleaved in the list? or if the GIL is removed does the interpreter segfault (I'm pretty sure this latter will not be the case for GIL removal but I don't understand the gil remove plan very much yet).

Edit: before people jump in and correct how the above is a bad idea anyway, it's not like I'd ever do the above and expect anything but disaster. This is more of a thought experiment to understand what GIL removal is going to do.

[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by colesbury on Friday November 17, 2023 @10:04AM (#38304522)
You will always get either [1,2,3,4,5,6] or [4,5,6,1,2,3] in the upcoming `--disable-gil` builds of CPython 3.13 and the nogil forks. Most operations on mutable collections hold a per-object lock.

Part of the integration work will be to better document the thread-safety guarantees, but there is still a lot of work to do before we get there.

[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by HDThoreaun on Friday November 17, 2023 @10:25AM (#38304844)
Every mutable collection needs to lock before write? Sounds slow af
[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by quietbritishjim on Friday November 17, 2023 @11:08AM (#38305462)
Locks are generally very cheap if they're not contended. Of course it's all relative though!
[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by gnulinux on Friday November 17, 2023 @02:13PM (#38308506)
They're cheap but they're not free. If you do it at literally every single rw operation at runtime, it's going to add up.
[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by fbdab103 on Friday November 17, 2023 @11:17AM (#38305596)
Last I looked, the nogil implementation was some 5-10% slower than the current, owing to all of the extra locking.
[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by OskarS on Friday November 17, 2023 @09:49AM (#38304347)
I'm guessing interleaving is probably not possible because the `extend` call is passed the full list, and it holds the GIL until it's finished. But you'd have to look at the bytecode to be sure, I suppose. If there were three append calls instead of a single extend, I would assume any interleaving would be valid.

> or if the GIL is removed does the interpreter segfault (I'm pretty sure this latter will not be the case for GIL removal but I don't understand the gil remove plan very much yet).

I haven't looked into the plan in detail either, but presumably not, that would be nuts. My understanding is that they're going to replace the GIL with locks on the objects themselves (your list `l` in this case). This is why in all the tests single-threaded performance suffer, you have to take and release more locks if you don't have a GIL, and the objects themselves grow larger as well.

[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by Gabrys1 on Friday November 17, 2023 @12:29PM (#38306820)
Let's say you're passing 1 million ints to each extend. One may start wondering if those operations are then split into chunks...
[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by dfox on Friday November 17, 2023 @09:58AM (#38304419)
You can probably trigger the behavior of random interleaving even with GIL if you use something other than list or tuple as an argument to list.extend(). CPython special cases list and tuples and copies the contents directly (list_extend_fast() in listobject.c) while it uses iterator for other types (list_extend_iter()).

Because the iterator can be pretty much arbitrary Python code it seems to be a bad idea to guarantee extend() to be atomic, as you don't want to hold the list mutex while calling out into user code.

[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by nerdponx on Friday November 17, 2023 @10:16AM (#38304705)
All the more reason to wonder. What's going to be special-cased to become atomic and thread-safe, and what won't be?

A surprising amount of the CPython stdlib is just pure Python code.

[ Parent | on HN ]
Re:The Changing "Guarantees" Given by Python's Global Interp... by ngoldbaum on Friday November 17, 2023 @09:44AM (#38304284)
It’ll be guarded by a fine-grained mutex, so it won’t seg fault. They’re using a performant mutex based on webkit’s wtf::lock.
[ Parent | on HN ]