Things don't get lost. You can requeue messages that end up being dead-lettered due to bugs. You can scale the number of services that read from that queue. You can have services up independently if they just read and write from a system that says, "here's a chunk of data to process."

You can loose data with Rabbit, and if you're starting from the ground up you might want to look at something like Kafka instead. If you don't care about message loss and just need throughput (like if you're doing real time logs or sensor data), something like zeromq might be more appropriate.

The idea behind microservices is that you gain stability at the expense of some redundancy.

We also split the messages into different types, commands and events. A command is to perform a specific action and is normally aimed at a specific service. And event alerts any subscribed service that a action has happened.

So you send a CreateOrder command. After processing it would fire off a CreatedOrder event which any subscribed service could pick up(Such as a EmailService sending a confirm email). Allows you to create workflows, and monitoring systems quite easily.

I'm interested in your experience with RabbitMQ?

I'm not directly on the team that maintains our RMQ instances so I'm not the best to comment on it. Our dev-ops team have a complicated set of shovels and federation to deal with/minimize data loss, but I haven't had to delve into those scripts too much yet.

You certainly can do it that way, and there are lots of benefits, but queues (and if you're doing this at scale, distributed queues) are an entirely distinct subject matter that I chose not to touch on for this.

Also, things can still get lost with queues. Network failures, partitions, bugs in your own publishers and consumers... nothing is fool-proof, and it takes a lot of work to reduce that loss rate to 0.

I find this an odd juxtaposition. On one hand I see what you are saying, but on the other I believe the nature of MS architecture places huge emphasis on State and State management that isn't necessarily as large a focus in Monolithic patterns. Stateless versus Stateful and now the creative application of "Soft"-state design patterns become non-trivial concerns in a MS world.

To that end, I feel like some address (positive or negative) on the impacts MS architecture have on state and the pros-and-cons I think are topical.

I was also wondering, for my own education, if you could expound a little on your points here:

> Also, things can still get lost with queues. Network failures, partitions, bugs in your own publishers and consumers

Specifically:

> Network failures

I had always thought that the delivery guarantees of TCP (and maybe the new QUIC protocol over UDP) were robust against network failures. What kind of edge cases would result in data loss in queue context?

> partitions

I am entirely ignorant on how partitions might result in data loss...so what kind of things have you seen?

> bugs in your own publishers and consumers

Isn't this true across any application regardless of architecture? Or are you implying that MS architecture/Queues some how increase the likelihood of bugs by nature?

Thanks in advance! (I always enjoy learning new things and hearing about/learning from other's experiences)

Just because TCP provides resilience doesn't mean that you're perfectly defended against all kinds of issues here.

I'm mostly talking about things in the category of bugs in the software layer using TCP (the drivers, the consumers/publishers, race conditions, batching messages, not receiving ACKs, etc). There are a lot of little things that can go wrong.

In terms of partitions, look up a series by a guy named Aphyr called "Jepsen". It goes over the CAP theorem as it applies to distributed datastores and queues. His examples and tests will demonstrate the concepts behind partitioning better than I can explain over a HN comment :)

And yes, these types of failures are implicit everywhere - but every additional layer you add, every hop in the chain, every interaction added to the request flow increases the surface area for problems. Especially once you push high scale with hundreds of nodes, become nic or cpu bound, etc etc.

There is a lot to unpack here, and it's not as simple as it seems on the face of it.

While these issues are definitely there, the problem of understanding a subsystem also exists on large monolithic applications. The not-my-problem syndrome seems like an indicator of bad devs and/or company culture (I've seen this happen once) and might be mitigated by making people work on multiple microservices instead of completely separating them.

In my opinion, separating a big project into multiple services emerges from the fact that most languages don't have built-in features for modularization on a large scale. In Java, for instance, there are methods < classes < packages. Once there are dozens of packages and thousands of classes, a project becomes hard to navigate and understand, especially when code isn't well-written with hacks disregarding modularizations. Using microservices reduces scope and prohibits some of these no-gos.

At that point you step up to using multiple maven modules. Which can give much the same development experience as microservices, but without the extra network boundary.

However, since all code is compiled into one large application there are other problems arising.

One I've seen often is sloppy dependency management: A explicitly depends on B, B on C. Now, a dev working in A starts using classes from C without declaring the dependency. Everything works because C is still included through B, but as soon as B gets removed, things start to break. There is mvn dependency:analyze to discover such issues, but it's not 100% exact, and people need to actively use it.

I think such a system built into and enforced by a language would be another quantum leap.

But yeah, at least it fails on compile time already.

Just go to search.maven.org and do a full class name search, no?