News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pwaivers on Friday February 01, 2019 @10:44AM (#19054988)
The overload indicators are strange to me. For example, one indicator is "IO thread idle below 20%: this means the pool of worker threads used by kafka for handling client requests are too busy to handle any more workload". How did they determine this? Is it true?

I think a much better way would be to monitor the JMX reporter for dropped messages or high latency: https://docs.confluent.io/current/kafka/monitoring.html.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pram on Friday February 01, 2019 @03:03PM (#19057933)
There are JMX beans for the socket server and request handler idle. If they even dip below 80% you’re probably looking at poorly configured producers timing out.