News for hackers. Stuff that matters.
Re:Kafka Is Dead, Long Live Kafka by derefr on Monday August 07, 2023 @01:50PM (#37037996)
Is there a reason you built that cache layer yourself (rather than each node "just" running its own sidecar MinIO instance, that write-throughs to the origin object store?)
Re:Kafka Is Dead, Long Live Kafka by richieartoul on Monday August 07, 2023 @02:04PM (#37038198)
(WarpStream co-founder)

The cache is for reads, not writes. There is no cache for writes.

We built our own because it needed to behave in a very specific way to meet our cost/latency goals. Running a MinIO sidecar instance means that every agent would effectively have to download every file in its entirety which would not scale well and would be expensive. We also have a pretty hard and fast rule about keeping deploying WarpStream as simple as rolling out a single stateless binary.