News for hackers. Stuff that matters.
Re:Celery 4.0 by parhamn on Saturday November 12, 2016 @07:00PM (#12940945)
Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficulties debugging and tracking exceptions, poor monitoring tools like 'flower' etc. all lead to this. Problems were exacerbated by the fact that simple async operations in python (which are easily fixed in more concurrent languages with a simple go func{}()) end up requiring a heavy distributed solution like Celery (or the lighter RQWorker) which creates a whole host of issues.

I imagine as native async tooling improves in py3x (async/await, aiohttp, and other tools) use of celery to do trivially concurrent things will decrease and Celery's usage will focus on more complex workflows (chords, fanouts, map/reduce).

Looks like many concerns we're tackled here (thanks Celery team) and I'm looking forward to playing around with this release.

Re:Celery 4.0 by andybak on Saturday November 12, 2016 @07:31PM (#12941084)
I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years.

I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that are more than good enough for most.

Re:Celery 4.0 by asksol on Saturday November 12, 2016 @07:57PM (#12941184)
Celery was never meant as a replacement for cron, it was simply a nice bonus that fits the messaging pattern well. Writing a task queue is actually very simple using for example Redis, but that doesn't necessarily mean Celery is over-engineered IMHO. It's very easy to forget the support required once your system is in production.

Disclaimer: I'm a contributor

Re:Celery 4.0 by k_sze on Saturday November 12, 2016 @08:59PM (#12941416)
By "contributor" you mean you are the main author, right? XD

Joking aside, we use Celery in production for asynchronous number crunching. We build a web UI in Django that raises Celery tasks that run long number crunching jobs. When the number crunching is done we look at the report through our Django web UI.

It works well enough, though we wish there were better built-in task management. E.g. A built-in API to reshuffle tasks (e.g. for I/O resource balancing) would be nice. celery purge also seems too drastic in purging every task in every queue. Or maybe we're just doing it wrong.

Re:Celery 4.0 by andybak on Saturday November 12, 2016 @08:00PM (#12941196)
I'm not saying Celery is over-engineered in general. It's just over-engineered in the context I've often seen it recommended in. i.e. for people learning or people with fairly modest requirements.
Re:Celery 4.0 by collyw on Sunday November 13, 2016 @01:08PM (#12944628)
Yep, I first asked on Stack overflow the best way to achieve a background task in a Django app, and the answers all said Celery. Considering it got run around once a week, it was a very over engineered solution I ended up with.

Its useful to know Celery, and gets used in a proper context in my current work, so I guess learning it wasn't a waste.

Re:Celery 4.0 by rtpg on Saturday November 12, 2016 @09:02PM (#12941428)
I think the advantage with tools like celery is that it deals with the many different failure scenarios pretty well.

It's kind of like jQuery's ajax. Of course you can figure out how to make a simple replacement. But then you have to manage the 100 different edge cases for when your code doesn't go down the happy path.

Much easier to write a simple task queue that's good enough than a replacement for $.ajax, though...

Re:Celery 4.0 by jjawssd on Saturday November 12, 2016 @10:38PM (#12941725)
Well said
Re:Celery 4.0 by gtaylor on Sunday November 13, 2016 @01:21AM (#12942130)
For low and medium traffic sites, the simplest usage patterns are really easy: http://docs.celeryproject.org/en/latest/getting-started/intr...

The best thing about celery is that you don't have to use any of the advanced features. If you need the basics, stick to the basics.

On the flipside, if and when you do need more than the basics, your system will grow with you. No need to hack up cron monstrosities as you grow beyond a single server. Though, cron vs celery is kind of an apples to... celery comparison.

Re:Celery 4.0 by andybak on Sunday November 13, 2016 @05:17AM (#12942677)
Even if the getting started is simple, there's still a ton of code and dependencies you're bringing into your project. Most importantly you've suddenly got dependencies on persistent processes such as Redis and RabbitMQ which need to be installed system wide. That's something that needs to be factored in to deploys, configs, restarts etc. You now need new tutorials for every deploy method (Heroku, Webfaction, PythonAnywhere and other outliers).

Yeah - everyone should ideally be comfortable with all this stuff but I try and keep anything that's more complex than a pip install in a virtualenv to an absolute minimum.

Re:Celery 4.0 by bar-b-q on Sunday November 13, 2016 @07:00AM (#12942940)
When building a small web app a few weeks ago, I tried to avoid additional dependencies, too. I was a bit stuck when I need to send an email asynchronous. Do you have a _simple_ recommendation to solve this without pulling something huge like celery+redis in to solve this task?
Re:Celery 4.0 by andybak on Tuesday November 15, 2016 @05:39AM (#12956998)
Depends on what you need. The bare minimum is to just spawn process so the request can return and your email can send. You tend to lack much control of failure conditions that way so you'd want to have some code in in the normal request/response cycle that checks for success or failure and informs the user.

You could add your emails to a db table and have a cron job consume them.

But for sending an occasional email? I've never really had a problem with just connecting to the SMTP server in the request/response cycle. If it takes more than a second to send then something is seriously wrong. You could use Mailgun or similar services which have their own queue for handling bulk sends and further reduce the likelihood of a problematic blocking of the web-server.

Re:Celery 4.0 by bar-b-q on Wednesday November 16, 2016 @03:45AM (#12965397)
I'm in a corporate network where sending the email over our internal smtp takes about 5 seconds. I dont want the user to wait for this delay, so I'm using celery currently. It just felt a bit too much for such a little task.

Maybe I will try something simpler like http://stackoverflow.com/a/4447147

Re:Celery 4.0 by brianwawok on Sunday November 13, 2016 @08:20AM (#12943174)
RQ + Reddis is "easier" http://python-rq.org/

If you are on AWS, you could just use something like their email service to fire off the request, then you don't need a queue at all.

Re:Celery 4.0 by rch on Saturday November 12, 2016 @10:54PM (#12941771)
For little things, it doesn't matter much what you do, and for bigger things I've moved to Luigi for background tasks.
Re:Celery 4.0 by ceronman on Saturday November 12, 2016 @09:53PM (#12941575)
"Starting from Celery 5.0 only Python 3.5+ will be supported."

This is a small detail, but I'm glad that one more big Python project is dropping 2.x support. The 2/3 split is not good for the community.

Re:Celery 4.0 by alexhayes on Sunday November 13, 2016 @03:22PM (#12945432)
Agreed!
Re:Celery 4.0 by ben_jones on Saturday November 12, 2016 @06:26PM (#12940794)
IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding.

Edit: Apparently they curtailed some features for simplicity as well.

Re:Celery 4.0 by asksol on Saturday November 12, 2016 @07:26PM (#12941051)
Thank you for the kind words, it's so very appreciated :)

I have merged many features, like broker transports, result backends, etc, and while the initial contribution was great, it ends up being unmaintained with issues that nobody fixes.

If there's any feature that you really want back, chances are the problems with that feature are not super difficult to fix, so please reach out!

Re:Celery 4.0 by ergo14 on Saturday November 12, 2016 @07:46PM (#12941142)
Asksol we use celery at AppEnlight extensively and love it. Thank you for your great work.
Re:Celery 4.0 by hueving on Sunday November 13, 2016 @02:28AM (#12942272)
Do you have a low-hanging fruit tag for new contributors to start with?
Re:Celery 4.0 by Buttons840 on Saturday November 12, 2016 @06:51PM (#12940910)
Some have the opinion that Celery has (or had) too many features. I know I had to twiddle with a lot of settings in Celery for my unusual use case, but it worked well in the end. There was a long standing bug which ended up being my misunderstanding a certain setting. All that said, I would use Celery again.
Re:Celery 4.0 by brianwawok on Saturday November 12, 2016 @06:38PM (#12940847)
Better to have 10 good features vs 20 bleh ones right?
Re:Celery 4.0 by stuaxo on Saturday November 12, 2016 @09:51PM (#12941572)
On my last project I had to use Celery. In the end, I had to look inside the source... having done the same with Python + Django I was apprehensive.

In the end I shouldn’t have been, it was a pleasant surprise. It's great 4.0 has come out.

Almost every Django developer will touch celery at some time, if your organisation can support development please do, it's not just an important piece of software, but well put together too.

Re:Celery 4.0 by cwyers on Sunday November 13, 2016 @12:11AM (#12941966)
> Nowadays it’s easy to use the requests module to write webhook tasks manually. We would love to use requests but we are simply unable to as there’s a very vocal ‘anti-dependency’ mob in the Python community

I'm not a heavy Python user, and I've never heard this before. It sounds... less than good.

Re:Celery 4.0 by ryankask on Sunday November 13, 2016 @04:25AM (#12942551)
The author may be referring to libraries themselves having dependencies. These attitudes are changing. For example, last week the Django community published a draft proposal that declares that "Django can have dependencies". The "Background" section is a good read and helps explain the origins of these attitudes: https://github.com/django/deps/blob/master/draft/0007-depend....
Re:Celery 4.0 by gtaylor on Sunday November 13, 2016 @01:23AM (#12942136)
It's not universal. I'd rather bring in a dependency than reinvent the wheel.

requests is used so widely that I'm surprised to see this raised as an issue.

Re:Celery 4.0 by fuhrysteve on Saturday November 12, 2016 @08:51PM (#12941384)
Congratulations to @asksol and the rest of the team for sealing the deal on 4.0! I've been waiting for a number of the features now in 4.0 for a long time now. I know you guys have been busting your asses for a long time and juggling with some complicated dependencies along the way.

Some time ago I built a little celery addon library as a sort of experimental way to solve the problem of having dynamic celery beat scheduled tasks. I never ended up implementing anywhere for a few different reasons: https://github.com/fuhrysteve/CeleryStore

I really like the concept behind the old djcelery project. But I don't use django much these days, and I'd like for it to be more compatible with tools I've become more familiar with (sqlalchemy / etc).

Do you have any advice for how to approach this? I know 4.0 introduced some new abilities to add beat entries via API.

Re:Celery 4.0 by welder on Sunday November 13, 2016 @07:54AM (#12943104)
In case you also run into the depreciation of sending error emails[1] I've had to switch back to using a decorator[2] to catch unhandled exceptions and generate error emails.

[1] CELERY_SEND_TASK_ERROR_EMAILS config removed http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#fe...

[2] https://gist.github.com/alanhamlett/dc8cdd4721ea63053f14#fil...

Re:Celery 4.0 by kkirsche on Saturday November 12, 2016 @06:35PM (#12940830)
Just started using this for the first time. It's pretty nice. I'm still trying to figure out how to really use it with Flask/Connexion cleanly and not in the top level module, but it's made my life much simpler for handling long running tasks (e.g. When needing a status 202 response). Great project. Sad to see SQLAlchemy was removed from the brokers though I understand.
Re:Celery 4.0 by brianwawok on Saturday November 12, 2016 @06:37PM (#12940842)
You don't want to use a DB a broker.

For AWS they have SQS which is awesome. Now if someone would write a Google cloud pub/sub, all would be well in the world.

Re:Celery 4.0 by pdpi on Saturday November 12, 2016 @06:40PM (#12940857)
Hum, Google Pub/Sub seems roughly equivalent to SQS to me?

https://cloud.google.com/pubsub/docs/overview

Re:Celery 4.0 by brianwawok on Saturday November 12, 2016 @07:19PM (#12941026)
Yes but no one wrote the kombu broker for it yet to my knowledge. To use a new broker in Celery you have to write code on how to send and receive messages.
Re:Celery 4.0 by sandGorgon on Sunday November 13, 2016 @01:57AM (#12942202)
Has anybody compared Celery 4 to rq (which started off as a much simpler+performant alternative to celery).

Would love to see how they stack up after this release.

Re:Celery 4.0 by asadjb on Sunday November 13, 2016 @02:38AM (#12942303)
We recently moved one of our services from Rq to Celery. We are using an older version of Celery, but this comment should still apply. While Rq is a great way to go in the start, if you have a large number of messages you are handling, you'll get into memory bottlenecks. This isn't a problem with Rq, but with using Redis as a broker. So if anyone is considering Rq vs Celery, they should keep this in mind.

The reason we switched to Celery was the volume of messages we were handling. Since Rq relies on Redis, all your messages need to fit in memory. While Rq was great and simple to setup at start, as we grew we were consistently dealing with Rq breaking because Redis was full and stopped accepting any write operations.

We moved to Celery because it could use RabbitMQ as a broker. RabbitMQ offloads most messages to the disk which has nicely taken care of the memory limitation issues.

With Rq we would get stuck after 10K messages (our messages included images so individual message size was large). With RabbitMQ I've seen the queue grow to about 120K without so much as a single hiccup.

Re:Celery 4.0 by sandGorgon on Sunday November 13, 2016 @03:02AM (#12942375)
That's very insightful. Quick question, since you are doing this in production - how are you serializing images into a message ? Base64 or something else.
Re:Celery 4.0 by zo1 on Sunday November 13, 2016 @06:54AM (#12942925)
Not the person you're asking, but from the experiences I've had with Celery + other messaging queues: Don't pass around large binary blobs if you can avoid it. Whether that is an image or something else is irrelevant.

You can do such things as passing a database unique key, GUID, or file-path to the raw data on disk. Obviously, you will also need to engineer around that if you've got a distributed system. The tangential benefit is that you're not using a "messaging" queue or system for persisting or semi-persisting your image data. That's a big no-no as such systems are transient in nature and that often doesn't align with binary or image processing.

Base64 is for when you want to pass around binary in a text-based format. E.g. XML or JSON. But do keep in mind that because of the encoding format, converting binary data to base64 does increase the payload size by about 15% or so.

Re:Celery 4.0 by asadjb on Sunday November 13, 2016 @03:08AM (#12942387)
We just do a str(file.read()). Not the most elegant solution. :)
Re:Celery 4.0 by brianwawok on Sunday November 13, 2016 @08:23AM (#12943187)
You really should not ram images in your task queue. Send them off to S3 or Cloud Storage with a GUID, and send along the GUID however you want (JSON is fine)
Re:Celery 4.0 by fuxi on Sunday November 13, 2016 @07:28AM (#12943015)
I have been using celery for years now and I love it. I was excited to see 4.0 with SQS support. However after giving it a spin I run back to 3.x.

Unfortunatelly version 4.0.0 has too many bugs for now. I will need to check back in few months. Better tests and better mocks would prevent the issues.

Re:Celery 4.0 by tschellenbach on Saturday November 12, 2016 @10:41PM (#12941732)
Many thanks Asksol, beautiful software. Have been a happy user for many years now. Hope to try out 4.0 soon!
Re:Celery 4.0 by ledil on Sunday November 13, 2016 @08:22AM (#12943184)
can you suggest a monitoring tool for celery? I want to view what tasks are currently working... is flower the only solution? thx
Re:Celery 4.0 by alexhayes on Sunday November 13, 2016 @03:27PM (#12945454)
Have you tried running 'celery events' on the command line? I find this sufficient in most cases.
Re:Celery 4.0 by ledil on Sunday November 13, 2016 @07:09PM (#12946575)
should I enable something to activate celery events?
Re:Celery 4.0 by gcb0 on Saturday November 12, 2016 @10:09PM (#12941625)
a rabbitMQ client? what's so awesome that everyone here says it is essential for python web projects? Honest question, never used python for that.
Re:Celery 4.0 by pandler on Sunday November 13, 2016 @01:24AM (#12942141)
I haven't used Django in a while, but, if I'm not mistaken, it's basically the easiest way to add any kind of asynchronous behavior to a python web server. There are few other options, and I don't think any of them are as mature as Celery.
Re:Celery 4.0 by zo1 on Sunday November 13, 2016 @06:57AM (#12942936)
The python Tornado web server does have the ability to do async tasks. However, it'd better be non-blocking otherwise your nice async web-server is going to stop serving requests while your task is processing.
Re:Celery 4.0 by logronoide on Sunday November 13, 2016 @04:55AM (#12942605)
Worst experience with a python toolkit ever. I hope new versions fix all bugs and issues.
Re:Celery 4.0 by alexhayes on Sunday November 13, 2016 @03:34PM (#12945496)
Of course, all other software is bug free...

Honestly this is a complex problem area and I think the Celery developers have done an excellent job of making it pretty trivial to get up and running while providing lots of flexibility for more advanced users! Not an easy feat.

Are there bugs? Of course - but I've never come up against one that I can't work around. Is that annoying? Sometimes but that's software development.

Re:Celery 4.0 by sateesh on Sunday November 13, 2016 @07:42AM (#12943065)
Being a user of celery, I find this comment unfair.Adding some points about why you consider Celery to be bad will help.