News for hackers. Stuff that matters.
Re:Ask HN: Whose building on Python NoGIL? by ptx on Monday December 23, 2024 @10:48AM (#42495217)
Using multiprocessing adds some overhead for because of serialization, so it's slower than it would be to just hand off the Python objects directly to the workers (as you can with threads) because the process doing the parsing also has to spend time on serializing them again. So you can avoid the GIL, but it has a cost.

For example, if I parse an XML document with ElementTree, as a quick experiment, parsing the document takes ~1 second and serializing all the elements names and attributes to JSON takes an additional ~0.5 seconds. Serializing the whole ElementTree object using pickle takes ~4 seconds. Serializing it as XML takes roughly as long as parsing it.

Re:Ask HN: Whose building on Python NoGIL? by dec0dedab0de on Monday December 23, 2024 @12:47PM (#42496164)
multiprocessing has more overhead even without serialization. I just brought it up to expand on why the GIL would force someone to thinking about being data-parallel.
Re:Ask HN: Whose building on Python NoGIL? by ptx on Monday December 23, 2024 @01:11PM (#42496347)
What I was trying to say, I guess, is that the additional serialization overhead can't be parallelized, which means that parallelization with multiprocessing doesn't help much in some cases where GIL-free threading would.
Re:Ask HN: Whose building on Python NoGIL? by tgma on Wednesday December 25, 2024 @02:47AM (#42507542)
Plus, [de]serialization often needs to be done under GIL (although under non-shared GIL per process, it interferes with any other async thing that may be done in each process).

It also screws up C extensions that don't support fork and does not solve the problem of increasing TLS handshake/fanout in outbound requests by the number of cores.

The whole thing sucks in production.