Re:Introduction to Event-Driven Architectures with RabbitMQ
by rmetzler on Thursday August 29, 2019 @12:11PM
(#20830786)
The bug description sounds a little bit like a memory leak or connections aren’t closed properly.
Re:Introduction to Event-Driven Architectures with RabbitMQ
by noncoml on Thursday August 29, 2019 @12:43PM
(#20831144)
This. You are leaking connections probably.
Re:Introduction to Event-Driven Architectures with RabbitMQ
by iddqd on Thursday August 29, 2019 @12:14PM
(#20830831)
You might be better off focusing on improving fault tolerance instead of trying to hunt down the root cause.
Re:Introduction to Event-Driven Architectures with RabbitMQ
by eternalban on Thursday August 29, 2019 @01:15PM
(#20831527)
Rabbit supports the choice of durable exchanges, queues, and messages. There are (tunable, negotiable) performance penalties in terms of latency and throughput.
We are partly in agreement regarding f/t. Logging comes into play when digging into recurring failure (bugs). Fault tolerance will not save you from bugs and bug hunting.
[p.s. to add, performance tuning/trouble-shooting also require instrumentation. Instrumentation would appear to be a fundamental (and useful) capability.]
Re:Introduction to Event-Driven Architectures with RabbitMQ
by itronitron on Thursday August 29, 2019 @12:57PM
(#20831317)
it sounds like this setup has proven very successful in helping your predecessor get hired (away from the company), I wonder if that was an implicit or an explicit requirement when they were designing the system
I'm currently working on solving a problem with an insanely over complicated setup (for the task at hand) that is build by another engineer who has since left the company.
It's a cluster of 3 virtualised machines running docker swarm where a RabbitMQ instance ties 40+ worker pods together. Once every 5 days or so the connection between RabbitMQ and (some of) the worker container stops working, causing the worker to crash and the queue message is lost.
We are talking like 5 layers of virtualization and/or abstraction. It's impossible to debug. I honestly don't know how to explain this to my customer.