Ultra-low-latency, batching and concurrent queue for IPC in Java

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by capmorganbih on Wednesday November 27, 2024 @06:05PM (#42260698)

Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by kasey_junk on Wednesday November 27, 2024 @06:40PM (#42260939)

You could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by capmorganbih on Wednesday November 27, 2024 @06:50PM (#42261015)

Could you please suggest java library for this if you know one?

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by kasey_junk on Wednesday November 27, 2024 @07:03PM (#42261069)

https://github.com/OpenHFT/Chronicle-Queue?tab=readme-ov-fil...

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by avinassh on Thursday November 28, 2024 @12:07AM (#42262633)

crazy idea, but SQLite

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by sriram_malhar on Wednesday November 27, 2024 @06:36PM (#42260903)

Can you explain what your issue is with Kafka? What makes it not reliable enough?

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by capmorganbih on Wednesday November 27, 2024 @06:50PM (#42261011)

Kafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by jojohohanon on Wednesday November 27, 2024 @11:50PM (#42262556)

There’s a whole conference in Vegas next week that want to sell you something.

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by dikei on Thursday November 28, 2024 @02:42AM (#42263341)

So basically, you want to build a write-ahead-log before writing data to Kafka, and I think you're underestimating the effort to implement a WAL.

If you don't have a person that can manage Kafka, you almost definitely don't have the person to maintain a WAL.

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by lizhenqi on Thursday November 28, 2024 @08:11AM (#42264994)

Just a suggestion, you can learn about automq (a message queue that has rewritten the storage layer based on Kafka), which has done a lot in automatic recovery and self-balancing, maybe it will help you. If you don't need it, you can ignore this comment.

[ Parent | on HN ]

Re:Ultra-low-latency, batching and concurrent queue for IPC ... by dingi on Thursday November 28, 2024 @12:02AM (#42262613)

Write your data to a MognoDB database. Then use the Kafka Mongo connector to pull data from MongoDB to Kafka.

[ Parent | on HN ]