News for hackers. Stuff that matters.
Re:Celery, non-blocking code and quest against coroutines by forgottenpass on Thursday June 09, 2016 @09:18AM (#11869140)
Design patterns are just that, design patterns. With upsides, downsides, suitable use cases and unsuitable use cases. I was originally going through pointing out all the nonsense that was outright wrong or just ruffled my feathers, but this whole article is punching above it's weight. And boils down to the following:

The author doesn't like using concurrent programing techniques in a single-threaded process.

The author does like a job queue that manages task execution across multiple processes.

This has fuck all to do with the GIL beyond the fact the presence of the GIL might have contributed to the popularity of the former in python and that celery uses multiprocessing over multithreading.

Designing a piece of software is more than cargo-culting the latest fads. Who cares if asyncio exists? Yes, we "really need" it to the extent that people want to design their programs like that. No, you don't "really need" to use it.

Re:Celery, non-blocking code and quest against coroutines by markbnj on Thursday June 09, 2016 @09:21AM (#11869155)
The title is off I agree, it's not like Celery fixes cpython's problematic data structures. I don't think it was a bad article, though.
Re:Celery, non-blocking code and quest against coroutines by forgottenpass on Thursday June 09, 2016 @09:33AM (#11869234)
I wouldn't have minded reading someones process of learning a new design pattern if it wasn't asserting so many things that are silly or flat out wrong during the process. That's why I originally wrote, "this is a bad article." I edified that bit out because it detracts from my point unless I take the time to go through and fisk the article.
Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @09:40AM (#11869279)
Well, I'm more about advertising the lockless programming pattern :)

Point out the silly, please, I like it when someone actually makes honest effort to teach me something.

Re:Celery, non-blocking code and quest against coroutines by scott_s on Thursday June 09, 2016 @10:15AM (#11869496)
The title, mainly. Most readers will take that to mean that you fixed the GIL itself - that is, removed it - rather than are presenting a pattern that avoids it. Your title sets up an adversarial relationship with such readers, because their attitude when they start is going to be "no, you didn't", which will cause them to miss your actual message.
Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @11:09AM (#11869873)
Fair point.
Re:Celery, non-blocking code and quest against coroutines by theseatoms on Thursday June 09, 2016 @09:53AM (#11869353)
I'm curious.. in what way are cpython's data structures problematic? Do you just mean as they relate to the GIL design choice, and the downsides/limitations that follow from the overall design?
Re:Celery, non-blocking code and quest against coroutines by coldtea on Thursday June 09, 2016 @10:12AM (#11869480)
He probably means the internal datastructures of Python the language (interpreter), not the datastructures it exposes to the user (list, map, tuple, etc).
Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @10:42AM (#11869675)
Yup. If you don't have to care about thread safety on API level, e.g. list, problems such as what's going on currently in gilectomy are no longer yours.
Re:Celery, non-blocking code and quest against coroutines by DasIch on Thursday June 09, 2016 @11:55AM (#11870218)
The GIL has atomic bytecodes as a side effect and this assumption has become part of the API.

There is nothing on the datastructure side that can be done to "fix" this.

In any case memory access should be considered a memory management problem just like allocating and freeing memory is. It needs to be solved on the language level to create sufficiently high level user experience.

You can only do that if you're willing to do things very differently such as Rusts Ownership/Borrowing approach.

Re:Celery, non-blocking code and quest against coroutines by markbnj on Thursday June 09, 2016 @11:07PM (#11874021)
Yes, this. Sorry I was away all day and didn't have time to reply.
Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @09:35AM (#11869247)
Hi, I'm the author. Please point out the nonsense :)

But honestly, what I'm trying to advocate here is that if we remove the problem of what's executing your tasks and the problem of implicit locking, we can stop worrying about GIL.

A very good implementation of that is how Grand Central Dispatch works.

Re:Celery, non-blocking code and quest against coroutines by martius on Thursday June 09, 2016 @10:52AM (#11869753)
Many examples have been commented already, but here are some of them:

> In the last 10 years I've never met a case where someone pointed out that "this code needs to be concurrent, but single threaded".

> Going back to concurrency - have I ever been constrained by the number of threads I can use? No, not even once.

Well, in the past 10 years, people have been working on the famous "c10k problem", about how a server software can handle a large amount of concurrent connections, since the model 1 thread/connection doesn't scale (the general purpose kernel scheduler will eat most of the CPU).

This is a real world problem.

> Do people use coroutines?

Yes they do, even if you claim they don't. I honestly don't know on which basis you can claim they don't.

> For me, this sums up the whole GIL and couroutines/asyncio debate. I think the main problem we have at the core of Python is that it's heavily inspired by C. But I think in this case it's Python's weakness.

I can't see a link between the GIL, C and coroutines.

One thing though, using an IO multiplexing framework when doing batch processing is probably suboptimal as using independent processes or queues will have the benefits you point: easier to read and understand, and less contention (kudos to my friend iksaif for the highlight). However, celery is a solution for some use cases, but will not work for many of them. For a similar queue-batch-work problem, executors can be a tool which fits someone needs better than celery (https://docs.python.org/3/library/concurrent.futures.html).

> I think that if we want to support full concurrency in Python, this is the way to go. Introduce primitives for fully lockless paradigm using queues and enable programmers to define queues.

Queues introduce head of line blocking (hence latency). You need several queues to handle independent and concurrent events in a timely fashion. This is why you need to multiplex.

> I think we never really wanted concurrent code, as programmers. I definitely never wanted coroutines and never needed to multiplex I/O.

Yes again, we do. That's how an interactive system works, that's how your computer works: one giant event loop handling concurrency between tasks and hardware inputs.

(update: words about concurrent.future executors)

Re:Celery, non-blocking code and quest against coroutines by danpalmer on Thursday June 09, 2016 @10:03AM (#11869420)
I was under the impression that GCD used green threads, which means it's sharing the same address space, and ultimately would be subject to the GIL?

When you're memory bound, sharing the address space is important, and Celery might not work well at all.

Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @10:07AM (#11869447)
Depends on how much memory we're talking about.

In regard to GCD - AFAIK it uses classic posix threads. In regard to GIL, if you can indeed guarantee that each task is atomic and lockless, you can use the same approach pyparralel uses

Re:Celery, non-blocking code and quest against coroutines by danpalmer on Thursday June 09, 2016 @11:16AM (#11869930)
I believe GCD actually multiplexes green threads across a pool of posix threads, which wouldn't work for Python because of the GIL.
Re:Celery, non-blocking code and quest against coroutines by richardwhiuk on Thursday June 09, 2016 @12:23PM (#11870400)
Correct. GCD has dispatch_queues, and each block on the dispatch queue will be executed on one of a pool of posix threads.
Re:Celery, non-blocking code and quest against coroutines by mdomans on Sunday June 12, 2016 @03:43PM (#11889900)
That's correct, but looking at the Python internals it seems that working around the GIL is possible if having this pattern exposed to end programmer.