News for hackers. Stuff that matters.
Re:Celery, non-blocking code and quest against coroutines by asksol on Thursday June 09, 2016 @12:07PM (#11870293)
Author of Celery here (not the article).

Celery solves a different problem: it's a distributed system that will help you run your tasks on multiple machines, not an alternative to asyncio/twisted/tornado etc.

The GIL is often easily worked around by starting N instances of your app, but that doesn't work for all applications (games, video, audio). Celery won't help you in those cases, but you can write C extensions. 99.99% of the time you shouldn't even consider using posix threads in Python, as most libraries (including popular ones) are not thread-safe, resulting in spending precious time fixing tricky bugs.

Celery is also not in contrast to coroutines, actually it's quite common to use Celery as a distributed layer on top of async I/O frameworks (top tip: routing CPU-bound tasks to a prefork worker means they will not block your event loop).

Another thing, the article says `scrape_url.subtask(args=(url,)),` is not very readable, but the idiomatic way to write this is: `scrape_url.s(url)` (yup we have a one letter method name, Django has Q we have .s). See more examples here: http://docs.celeryproject.org/en/latest/userguide/canvas.htm...

Re:Celery, non-blocking code and quest against coroutines by enjo on Thursday June 09, 2016 @01:37PM (#11870936)
Nothing useful to add other than thanks for the great library. I've been using Celery for years, it's made my life demonstrably better.
Re:Celery, non-blocking code and quest against coroutines by brianwawok on Thursday June 09, 2016 @01:03PM (#11870675)
TIL about the s method..

Thanks for the tip!

Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @03:44PM (#11871793)
In all honesty .subtask() is more readable for me than .s()

Also, I don't want Celery to replace asyncio and/or threading.

I was trying to point out that the programming model can be back ported to Python as a way of approaching concurrency, which, without having to support "locks everywhere" is simpler to implement with GIL

Re:Celery, non-blocking code and quest against coroutines by asksol on Thursday June 09, 2016 @05:23PM (#11872495)
I think it's more readable in isolation, but when you have workflows it distracts you from what it's actually doing, making it harder to see what the purpose is at a glance. An API can probably not have many shortcuts like this, but you learn what .s does once, and then you barely notice it.

But that's my opinion and I have come to learn not everybody value succinctness in code, so you have the choice of both :o)

Had I not been constrained by backward compatibility I may have made it so that task(arg) only defines the signature of a task invocation and you'd need to do task(arg).delay() to call it remotely, and task(args)() to call it as a function locally.

Re:Celery, non-blocking code and quest against coroutines by mdomans on Thursday June 09, 2016 @05:31PM (#11872545)
Not merely a choice of mine to make - working in a team actually makes .subtask better