Re:Why Was Apache Kafka Created?
by SJC_Hacker on Monday August 25, 2025 @06:20PM
(#45019823)
Out of curiousity what was the issue with just wrapping the AMQP-CPP pub/sub calls around a mutex?
Re:Why Was Apache Kafka Created?
by HarHarVeryFunny on Tuesday August 26, 2025 @09:01PM
(#45034244)
I'm a bit hazy on the full details because it was a few years ago, but basically it gets more complicated because you subscribe by installing an async callback which needs to ack/nak messages, needing locking, and will be called from the context of the network event loop that also needs locking. If you do any real work in the message processing callback then you'll be blocking the event loop, so the callback has to defer processing by queuing C++ lambdas capturing the context, and running those in a thread pool.
I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.
In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.