News for hackers. Stuff that matters.

Show HN: Kafka-compatible streaming on S3 with stateless brokers

Hi HN,

KafScale is Kafka-compatible streaming, k8s native, where S3 is the source of truth and brokers hold no persistent state. Written in Go, runs on Kubernetes.

Built this after years of operating Kafka and hitting the same walls: broker failures that take hours to recover, partition rebalancing that blocks deploys, disk capacity planning that never ends.

How it works:

- Producers and consumers use standard Kafka clients - Brokers buffer in memory, flush to S3 - etcd stores metadata and consumer group state - Recovery means restarting a pod and reading from S3 - Optional Iceberg processor reads segments directly from S3, bypasses brokers entirely for batch/analytical workloads

What you give up: latency is 400-500ms (S3 round-trip), no transactions, no compacted topics. It's not a 100% replacement.

What you get: brokers are disposable, scaling is just replica count, no disk management, direct access to streamed data over S3 ACL

License: Apache 2.0 GitHub: https://github.com/novatechflow/kafscale

( 2 comments | on HN )
Re:Show HN: Kafka-compatible streaming on S3 with stateless ... by stympy on Wednesday January 07, 2026 @10:02AM (#46527145)
Thanks for sharing this! I'd love to see a section in the readme comparing this to other s3-backed kafka replacements such as warpstream and bufstream, and why you'd use this over those.
Re:Show HN: Kafka-compatible streaming on S3 with stateless ... by 2pk03 on Friday January 09, 2026 @03:27AM (#46551344)
Did that here: https://kafscale.io/comparison/ - mainly k8s native, easy ops, direct decoupling of brokers and data streaming processing, 100% open source. Much cheaper to operate in a cloud.