News for hackers. Stuff that matters.
Re:What I wish someone would have told me about using Rabbit... by kelnos on Thursday July 14, 2022 @01:29AM (#32092363)
I get that distributed messaging/queuing is difficult (been there, done that, often didn't do a great job of it), but the constraint that every node in the cluster has to be running the exact same version of RabbitMQ is ridiculous. I can't see how you could ever orchestrate zero-downtime upgrades. Requiring that they're all on the same major version sounds reasonable, with a further constraint that clients can only use features supported by every node in the cluster (e.g. if a new feature was introduced in 1.3.0, and some nodes are still running 1.2.x, clients shouldn't use that new feature until all nodes have been upgraded). And there should still be some sort of reasonable migration process to the next major version! It may not be a simple migration process, but should at least be something where you can orchestrate things such that you have no downtime.

Having the default behavior during a network partition be "whatever, just chug along as if nothing is wrong" is bonkers. Yes, the person who first set up the cluster should have read the documentation and gone over the configuration file line by line to see what might need changing, but... damn, that's a terrible default. Sure, some people's applications might value availability over consistency, but that's not the safest choice that follows the principle of least surprise.

Using a higher-level library to interact with the cluster is really good advice, in general. We used Kafka at my last company, and colleagues who actually knew what they were doing wrote a (simple) wrapper library that set things up properly for our cluster so clueless people (such as myself) could write producers and consumers without having to understand what all the fiddly connection setup settings did, and how to handle various edge-case errors. Before that, quite a few outages were due to producer/consumer misconfigurations.

Also I can't imagine running this kind of infra on Windows servers. That sounds like a self-inflicted wound (by "self" I mean the company, not OP specifically). And the idea of Windows Update running on a prod server ruining your day... what? IMO infra should be as immutable as possible. Patching/updating software on a machine should be a matter of spinning up a new machine with an already-updated image (that you've built and tested elsewhere), bringing those new machines into a cluster, and then decommissioning the old ones. When colos and dedicated servers were all the rage, that was difficult (and sometimes impractical), but in this day and age, with on-demand cloud provisioning, there's no excuse for companies that can use that sort of infra.

Re:What I wish someone would have told me about using Rabbit... by code-e on Thursday July 14, 2022 @02:01AM (#32092551)
Support for rolling upgrades (one node at a time) instead of a full-stop upgrade was added in rabbitmq 3.8, about six months before this article was published. So while a full-stop is no longer necessary, it was likely a restriction for the majority of the time the author was using rabbit.
Re:What I wish someone would have told me about using Rabbit... by datavirtue on Thursday July 14, 2022 @07:49AM (#32094672)
"I can't see how you could ever orchestrate zero-downtime upgrades."

I didn't it before 3.8. Read the docs. All of them--three times.

Re:What I wish someone would have told me about using Rabbit... by pc86 on Thursday July 14, 2022 @01:49AM (#32092483)
> Patching/updating software on a machine should be a matter of spinning up...

While this is absolutely a "best practice" I really think there is a pretty big divide here between the best practice and what happens In The Real World (strictly by number of developers). Between $DAYJOB, consulting/freelancing, and the internet/tech twitter/HN/etc, I've seen basically three groups:

1. FAANG, startup, etc. that probably follows this best practice "cattle over pets" methodology more often than not. 90% of developers do not work here. For the same reason you can't compare FAANG salaries to really anything else, you can't compare their tooling and infrastructure either. They're playing a completely different game than most people.

2. Indie developers who want to mess around with this stuff.

3. Boring companies with IT departments where you have to request VM resources and wait for them to get spun up. I worked for a place within the last decade where the idea of getting a VM provisioned for you in less than 2 business days was laughable. This was a publicly traded, multi billion dollar company with probably 800 developers, IT, and support desk folks on staff and did not make any money from its software directly (we were not a profit center).

I am considerate of the "well you should just" mindset when it comes to dev ops and infrastructure, but most developers don't work in that environment. If the company has never had that sort of expertise or expense before, it's a hard sell especially if it's coming from a developer who has never worked in that environment before. And truthfully, the last thing I want to do as a software engineer is become "the devops guy" and now be responsible for the VM infrastructure of the entire team rather than actually writing code.

Re:What I wish someone would have told me about using Rabbit... by nightpool on Thursday July 14, 2022 @02:27AM (#32092696)
If you're not making any money from your software directly, that's even MORE reason that you shouldn't be self-hosting your own RabbitMQ clusters. Use a software provider that is stable, has already figured out all of the stuff, has an enterprise friendly SLA, and hold their feet to the fire. For the love of God, don't make your complicated production-critical ops work dependent on the weird organizational constraints of your backwards IT org. Plenty of those "FAANG" "startups" would be happy to sell you their expertise.
Re:What I wish someone would have told me about using Rabbit... by pas on Thursday July 14, 2022 @06:34AM (#32094183)
apparently they are not even making enough money to not-self-host things. enterprise? SLA? DRP? ha-ha. :/
Re:What I wish someone would have told me about using Rabbit... by vorpalhex on Thursday July 14, 2022 @01:33AM (#32092378)
You do zero downtime upgrades via cluster federation. Works great.
Re:What I wish someone would have told me about using Rabbit... by kelnos on Thursday July 14, 2022 @01:36AM (#32092391)
Ah, I see. I've never used RabbitMQ, so I'm not familiar. I guess OP didn't know about this either?
Re:What I wish someone would have told me about using Rabbit... by ikiris on Thursday July 14, 2022 @03:18AM (#32092986)
tldr: if you have a thing that can never go down, you need more of those things.