Now, this is what I'm hearing from what you are saying:
> 1- We can't use a JVM implementation because we aren't using a JVM language.
Makes sense.
> 2- The C library is okay, but hides its thread pool with no way to access it.
Ugh. Hate that. Its like these people writing these have never had to use them in a real project. The sign of a mediocre library.
Pony's actor model might have to rewrite almost any library used by it when concurrency is involved.
But now, I think you answered your own question in the titles now:
> Why we wrote our Kafka Client in Pony
1- Because the C library is mediocre and hides its threads from users making it not very useful for high-performance applications.
2- Because the rest of the system is in Pony. Really, you could write it in C/C++ or even Rust as long as you wrote it in a way that played well with Pony's concurrency model, but why bother with that extra effort, especially if you believe - as you seem to - that Pony's concurrency story is superior.
Also, while the C client is more featureful and better tested, there is still the concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources and would prefer to avoid that.
Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains. Yes, we mention in the blog post that we would be happy at being parity with the C client but our goal has always been to exceed it, eventually. Both in terms of performance and features.