Re:From Kafka to ZeroMQ for real-time log aggregation
by janczukt on Saturday September 17, 2016 @02:00PM
(#12521588)
This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.
It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.