Re:Liftbridge: Lightweight, fault-tolerant message streams
by tylertreat on Tuesday July 30, 2019 @10:40AM
(#20565203)
This is correct. Also, I'm planning to introduce a mode that runs NATS "embedded" within the Liftbridge server, such that NATS basically becomes an implementation detail of Liftbridge.
Re: consumer-offset checkpointing, this will be implemented in the form of consumer groups, similar to how Kafka works where a consumer group checkpoints its position in the log by writing it to the log itself.
If you require fanout to your stream consumers then this is supported simply by having multiple consumers subscribe to each stream. If you require a competing consumers model then this isn't directly supported, but you can have load-balance groups which you could use to achieve something similar, but with a static configuration - that is, your consumer population can't resize dynamically, but you can distribute messages to a fixed set of streams, each of which has a consumer.
liftbridge doesn't yet support durable consumer-offset checkpointing, although this is advertised as being on the roadmap. In the meantime, if you wanted the consumer offset to survive consumer restarts, then the consumer itself would need to take care of persisting its current offset.