Kafka Is Dead, Long Live Kafka

Re:Kafka Is Dead, Long Live Kafka by bothra90 on Monday August 07, 2023 @01:11PM (#37037393)

Exciting work! Some questions:

1. Any plans on open-sourcing this?

2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases.

3. Related to 2, how does WarpStream compare to tiered storage in Pulsar?

[Edit 1] Added (3)

[ Parent | on HN ]

Re:Kafka Is Dead, Long Live Kafka by smrtinsert on Monday August 07, 2023 @10:39PM (#37043816)

Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.

[ Parent | on HN ]

Re:Kafka Is Dead, Long Live Kafka by richieartoul on Monday August 07, 2023 @11:05PM (#37044024)

(WarpStream Cofounder)

FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t seem like anyone would want to be responsible for the metadata layer themselves. Would love your feedback on that.

[ Parent | on HN ]

Re:Kafka Is Dead, Long Live Kafka by banditelol on Tuesday August 08, 2023 @09:35AM (#37048762)

I'm also quite surprised there's no option for self hosting metadata on the free tier. To be fair, in my experience having managed metadata server for prefect (while they have similar Bring Your Own Server model) is quite hard to get right. But at the end we decided to keep maintaining it because our company prefer having all the data in our servers (including metadata).

But I'm still excited to try this, since at least now I could play around (and learn) with a partially kafka compatible system, without the burden of maintaining all of Kafka parts (and costs). Thanks!

[ Parent | on HN ]