The most blatant example is the countdown tasks. Celery has a very strange implementation of these (meant to be broker agnostic) where it consumes the task from queue, sees in the task custom headers (which is meaningless to RabbitMQ) that it should be delayed and then sits on the task and takes a new task. That results in heavy memory load on your celery client holding all these tasks in memory, and if you have acks_late set, RabbitMQ will be sitting on many tasks that are claimed by a client but not acked and _also_ have to sit in memory. But that is 100% a celery problem, not Rabbit, and we solved it by overriding countdowns to use DLX queues instead so that we could use Rabbit-native features. Not surprisingly, Rabbit performs a lot better when you’re using native built-in features.
I, too, would love an implementation of arbitrary-granularity delays without buffering (potentially huge, mem-wise) tasks in "unacknowledged" in a giant binary heap: that's expensive for consumers and terrifying for RabbitMQ stability when a broker has to e.g. re-"ready" millions of messages because a delay-buffer OOMed.
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...
The plugin keeps a node-specific database of the messages that are to be delayed. If the node is unavailable or lost, so too are the messages the node was keeping for future publish.
Also, there is no visibility in to the messages that are awaiting delay. 10? 100? 5000? Your guess is as good as mine and you'll never be able to figure out what's in to-be-published pipeline.
Signed, someone who has dealt with these issues.
The plugin keeps a node-specific database of the messages that are to be delayed. If the node is unavailable or lost, so too are the messages the node was keeping for future publish.
Also, there is no visibility in to the messages that are awaiting delay. 10? 100? 5000? Your guess is as good as mine and you'll never be able to figure out what's in to-be-published pipeline.
Signed, someone who has dealt with these issues.
This will probably not be a popular opinion here but the business of DoorDash is not to fix open source software.
I get your point. I use a lot of OSS in my business and we do contribute back. But not every business has the opportunity to observe the stack for days or weeks while the real clients suffer due to outages. Each client who is affected by an outage is most likely churning immediately and you're not going to win them back easily.
In essence, the migration to Kafka will only benefit them long term and they have ultimately made the right business choice.
It probably isn't. But as their business depends on software, it's worrisome that rather than understanding what was wrong, they just switched software. That doesn't seem inspiring to me.
Nobody said they had to fix the problems, but identifying them, and checking if they were fixed upstream seems like a good idea. Sometimes upgrading is a nightmare and doesn't fix anything, sometimes it fixes a major issue and drops cpu usage from 90% to 5%. If you know what the problem areas are, it should take less than a day to look at the changes and see if that area was addressed. Even if not, reading the changelogs and version notes should give some idea. It's fine if they didn't test upgrading, but again, I'm not inspired about their engineering talent and processes when it looks like they didn't even look to see if it would have been likely too help.
Engineering Daily podcast has various podcasts that come up around this. One that comes to mind was the one with Slack. Slack had hyper-growth, they went out and hired individuals who had already tackled these problems at the same numbers or larger.
You get hired at a company, and you look around at what FAANG is using, you implement that because you can always say, "But it works at Netflix". You build up immense expertise in the project, the tool, etc.
Switch jobs, ignore everything the existing company is doing, where the landscape is going and shove your agenda of implementing the same thing such as Kafka. Measure everything.
Look like the saviour of the company. Point to this, ask for massive stock compensation, bonus, etc. based on the metrics that make you look good.
1. Cache it's amazing... unless you need keys something more complex than key value, e.g. multiple keys, invalidate by pattern etc... So ... not so good actually.
2. Pubsub doesn't sale really good, and that'a a real limitation
3. The new pubsub is purely a copy of Kafka ( they admit that in docs) but in a much less supported, and feature rich version.
So despite I'm still using Redis in my Day today, as of 2020 I don't see redis in any new setup.
RabbitMQ has very good observability. It has built-in web dashboards as well as an API. Every day that I had to bang my head against the opacity of an org-wide Kafka or ActiveMQ setup at one job, I missed the simplicity and transparency of RabbitMQ from a previous job. Somehow Kafka earned the "trendy" badge though, so everyone just uses it by default.
This is unfortunate if you want to see the queues from up close.
I don't have the full picture, but I'd move to multiple brokers first, which seems like a low effort move, then away from Celery, in order to split worker code from the rest of the application. In the meantime, I'd instrument the hell out of Celery to see what was going on and push those changes back into upstream.
But the first stack I would reach for to handle a queue would be erlang/otp (or maybe elixir), is that not designed to handle large queues without failure?
Again I am absolutely no expert but I'd like to hear the views on this.
In absence of any significant stats its safe to assume their decision were driver by this two statements from article.
"There were no in-house Celery or RabbitMQ experts at DoorDash who we could lean on to help devise a scaling strategy for this technology."
"DoorDash had in-house Kafka expertise"
Option 1: Redis broker. Did they ever setup a test environment with a Redis broker to see how it compares to a RabbitMQ broker? Seems like they didn’t.
Option 2: Same as option 1. Did they actually try to setup a kafka broker for celery to test it out and see how it goes?
Option 3: Multiple brokers. They could have attempted a very simple setup starting with just two RabbitMQs and dividing the tasks roughly in half between them. This would allow scaling of RabbitMQ horizontally. At least try it and see.
Option 4: Upgrade versions. Did they even try setting up an environment with upgraded versions. They say there’s no guarantee it fixes their observed bugs. How about trying it out and doing some more observation?
Option 5: We decided to go all-in on this one without even trying the other 4 and doing any sort of comparison with actual tests or benchmarks. Maybe this was the best solution for them, and they chose wisely, but how do they know?
The big thing that gets me is they talk about the observed issue of celery workers that stop processing tasks as well as the limited observability of celery workers and RabbitMQ. These aren’t black boxes. They are open source. You can debug them yourselves, report the bugs, fix the bugs and submit patches, add features, fork it yourself if necessary, etc. The fact that they don’t know if new versions will fix their observed bugs makes it clear they never identified those bugs. What’s the point of running on an open source stack if you’re going to treat things like a black box anyway?