Re:Kafka Is Dead, Long Live Kafka
by derefr on Monday August 07, 2023 @01:50PM
(#37037996)
Is there a reason you built that cache layer yourself (rather than each node "just" running its own sidecar MinIO instance, that write-throughs to the origin object store?)
[ Parent
| on HN ]
Re:Kafka Is Dead, Long Live Kafka
by richieartoul on Monday August 07, 2023 @02:04PM
(#37038198)
(WarpStream co-founder)
The cache is for reads, not writes. There is no cache for writes.
We built our own because it needed to behave in a very specific way to meet our cost/latency goals. Running a MinIO sidecar instance means that every agent would effectively have to download every file in its entirety which would not scale well and would be expensive. We also have a pretty hard and fast rule about keeping deploying WarpStream as simple as rolling out a single stateless binary.