Again, scaling out to 30 nodes was not an option for us. It had to run on a single machine, unlike the rest of our data pipeline. While 200MB/s is nowhere near the theoretical limit of our hardware, it is near what Kafka’s stress testing tools achieved in the same setup, particularly with our record size (100 bytes).
I've worked with Elasticsearch clusters indexing close to a million documents per second from kafka. That takes a lot of CPU, memory, bare metal, etc. and it taxes the JVM garbage collection quite a bit. But it can be done. We basically maxed out CPUs across 30 or so nodes. Basically, when you get close to maxing out the disk IO, you know you are doing a decent job.
People reach for C for a lot of reasons. And not always the right reasons. But if it works, it works. A big reason seems to simply be that interfacing with it from various other languages is pretty straightforward. That's why python developers don't worry about performance. They can always plug in something native if it becomes an issue.