News for hackers. Stuff that matters.
Re:What If We Could Rebuild Kafka from Scratch? by bjornsing on Friday April 25, 2025 @12:25PM (#43795495)
> Key-centric access: instead of partition-based access, efficient access and replay of all the messages with one and the same key would be desirable.

I’ve been working on a datastore that’s perfect for this [1], but I’m getting very little traction. Does anyone have any ideas why that is? Is my marketing just bad, or is this feature just not very useful after all?

1. https://www.haystackdb.dev/

Re:What If We Could Rebuild Kafka from Scratch? by notfed on Friday April 25, 2025 @01:20PM (#43796214)
The website seems very vapid. Extraordinary claims require extraordinary evidence. Personally I see a lack of evidence here (that this vendor-locked product is better than existing freeware) and I'm going to move on.
Re:What If We Could Rebuild Kafka from Scratch? by bjornsing on Friday April 25, 2025 @02:36PM (#43797102)
Thanks for the honest assessment!
Re:What If We Could Rebuild Kafka from Scratch? by thesimon on Friday April 25, 2025 @01:57PM (#43796667)
> HaystackDB is accessed through a RESTful HTTPS API. No client library necessary.

That's cool, but but I would prefer to not reinvent the wheel. If you have a simple library, that would already be useful.

Some simple code or request examples would be convenient as well. I really don't know how easy or difficult your interface design is. It would be cool to see the API docs.

Re:What If We Could Rebuild Kafka from Scratch? by bjornsing on Friday April 25, 2025 @03:49PM (#43797870)
Yeah, it’s a bit of a chicken and egg problem. Since I don’t have away to find potential customers I feel it’s too risky investing in stuff like client libraries and good API docs. But I can definitely understand you’d like to see more.
Re:What If We Could Rebuild Kafka from Scratch? by galeaspablo on Friday April 25, 2025 @12:51PM (#43795875)
Some input from previously working on a superset of this problem. And being in a similar position.

Mature projects have too much bureacracy, and even spending time talking to you = opportunity cost. So making a case for why you're going to solve a problem for them is tough.

New projects (whether at big companies or small companies) have 20 other things to worry about, so the problem isn't big enough.

I wrote about this in our blog if you're curious: https://ambar.cloud/blog/a-new-path-for-ambar

Re:What If We Could Rebuild Kafka from Scratch? by bjornsing on Friday April 25, 2025 @03:06PM (#43797433)
Thanks. Interesting read, and an interesting product / service. Have been thinking about the same approach myself…
Re:What If We Could Rebuild Kafka from Scratch? by tinix on Friday April 25, 2025 @01:30PM (#43796362)
what can you do that redis can't?

I'm also skeptical of the graph on your front page that claims S3 cost as much as DynamoDB.

that alone makes it look like total nonsense.

as someone else said, extraordinary claims require extraordinary evidence.

Re:What If We Could Rebuild Kafka from Scratch? by bjornsing on Friday April 25, 2025 @02:52PM (#43797298)
> what can you do that redis can't?

Keep the data in S3 for 0.023 USD per GB-month. If you have a billion keys that can be useful.

> I'm also skeptical of the graph on your front page that claims S3 cost as much as DynamoDB.

Good point. Could have put a bit more work into that.

Re:What If We Could Rebuild Kafka from Scratch? by bjornsing on Saturday April 26, 2025 @12:32AM (#43800957)
> Good point. Could have put a bit more work into that.

On second thought, and after looking at my cost estimates, the reason DynamoDB ends up costing about the same as S3 for this kind of use case is storage costs. DynamoDB is a lot cheaper than S3 to write to, but 5-10x more expensive to keep data stored in. So after about 16-32 months you reach break even.