Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by pwaivers on Friday February 01, 2019 @10:44AM
(#19054988)
The overload indicators are strange to me. For example, one indicator is "IO thread idle below 20%: this means the pool of worker threads used by kafka for handling client requests are too busy to handle any more workload". How did they determine this? Is it true?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by pram on Friday February 01, 2019 @03:03PM
(#19057933)
There are JMX beans for the socket server and request handler idle. If they even dip below 80% you’re probably looking at poorly configured producers timing out.
I think a much better way would be to monitor the JMX reporter for dropped messages or high latency: https://docs.confluent.io/current/kafka/monitoring.html.