News for hackers. Stuff that matters.
Re:Hunting down a C memory leak in a Go program by xerxes901 on Saturday October 16, 2021 @12:53AM (#28885645)
So, in the interests of full transparency - we at Zendesk are actually running a fork of confluent-kafka-go, which I forked to add, amongst other things, context support: https://github.com/confluentinc/confluent-kafka-go/pull/626

This bug actually happened because I mis-merged upstream into our fork and missed an important call to rd_kafka_poll_set_consumer: https://github.com/zendesk/confluent-kafka-go/commit/6e2d889...

Re:Hunting down a C memory leak in a Go program by otterley on Saturday October 16, 2021 @12:55AM (#28885649)
Did you consider Segment’s pure Go library instead? Curious as to why you might have rejected it. (I should also note that Go C bindings impose performance penalties in many cases due to the need to copy memory back and forth.)
Re:Hunting down a C memory leak in a Go program by xerxes901 on Saturday October 16, 2021 @01:00AM (#28885669)
We had previously made extensive use of Sarama, another pure-go implementation of the Kafka protocol. After the third or fourth time where I had to use Wireshark to debug a protocol-level bug we figured it would probably be better to use the librdkafka-based library to have better protocol support.

Plus the team running our brokers found that the Sarama clients were a drag on them being able to enable broker features like zstd compression (although I understand Sarama has zstd now). This felt like it was going to be a kind of problem that would keep cropping up.

Re:Hunting down a C memory leak in a Go program by otterley on Saturday October 16, 2021 @01:01AM (#28885674)
You’ll notice Segment rejected Sarama, too. :-) Honestly I think only Segment got the pure Go library right. They care a lot about correctness and performance.
Re:Hunting down a C memory leak in a Go program by twmb on Saturday October 16, 2021 @01:06AM (#28885687)
I spent quite some time writing github.com/twmb/franz-go due to wanting more from both Sarama and from Segment’s library. Pure Go as well.

I’d recommend evaluating it as an alternative, even to confluent-kafka-go due to comprehensive feature support and (afaict) even higher performance than the aforementioned libraries.

Re:Hunting down a C memory leak in a Go program by xerxes901 on Saturday October 16, 2021 @01:15AM (#28885725)
Oh hey that looks really interesting. I'll certainly give it a look!

Lesson learned from the Sarama -> librdkafka migration: We actually wrapped Sarama in a home-grown abstraction (to handle some Zendesk-specific concerns). When we made v2 of this wrapper to support confluent-kafka-go, we made sure that no confluent-kafka-go implementation details/types leaked out of the interface. So, in theory, we can change underlying libraries again "easily" (well, with less rework in the apps at least).

Re:Hunting down a C memory leak in a Go program by twmb on Saturday October 16, 2021 @01:38AM (#28885798)
Nice! That’s definitely a good practice :).

I’ve tried to make the kgo package easier to use, but I think some libraries suffer from over abstracting the underlying protocol, which ultimately locks them out of some higher efficiency code or some correctness. So, I’ve tried to make the library simple yet not hide all of the details. It’s seemed to have panned out successfully for at least some people. Happy to answer any questions about it, and thanks for taking a look!

Re:Hunting down a C memory leak in a Go program by otterley on Saturday October 16, 2021 @01:08AM (#28885698)
Nice!