Re:Kafka at the Low End
by akshayshah on Wednesday February 19, 2025 @10:43AM
(#43103294)
Couldn’t agree more - the most exciting thing about KIP-932 is how much easier it’ll become to build a good HTTP push gateway.
Uber wrote a Kafka push gateway years ago, when it was considerably harder to do well: https://www.uber.com/blog/kafka-async-queuing-with-consumer-...
Do you mind explaining what you mean by not being able to find a "good Kafka client" for Ruby? There are pretty good bindings to librdkafka and frameworks like Karafka (https://github.com/karafka/karafka/) that provide many functionalities, including a Web UI.
You can over partition and make at-least-once workers happen (if you have a good Kafka client), or you use an http gateway and give up safe at-least-once. Hopefully this will make it easier to build an at-least-once style gateway that's easier to work with across a variety of languages. I know many have tried in the past but not dropping messages is hard to do right.