News for hackers. Stuff that matters.
Re:Redis Streams and the Unified Log by simonw on Wednesday November 08, 2017 @02:23PM (#15655981)
The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as many consumers as you like to reliably work through the log from any given position.

As a producer, you just need to know that your message has definitely been delivered to the log. What consumes it is none of your business.

Have you read https://engineering.linkedin.com/distributed-systems/log-wha... ? It's one of my favourite essays on software architecture of all time, and it goes deep into this concept.

Re:Redis Streams and the Unified Log by markbnj on Wednesday November 08, 2017 @02:35PM (#15656097)
> The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as many consumers as you like to reliably work through the log from any given position.

Right I think we're saying the same thing. In the article checkpointing was introduced to enable safely managing the size of the log, i.e. it appears that in this use case _somebody_ needs to know whether the consumers are caught up in order to adhere to at-least-once semantics. It just seemed to me that if you reach the point where you're rolling your own to solve that problem there are other possible solutions.

I haven't read the linked-in piece yet but I intend to. It was also linked in the article. Thanks for linking it here.