News for hackers. Stuff that matters.

A comment on Operating a large distributed system in a reliable way: practices I learned. Parent comment

Re:Operating a large distributed system in a reliable way: p... by packetslave on Wednesday July 17, 2019 @07:36PM (#20465153)
A single host stuck at 100% CPU also has a nasty effect on your tail latency, in a system with wide fanout. If a request hits 100 backend systems, and 1 of them is slow, your 99th percentile latency is going to go in the toilet.
[ Parent | on HN ]
Re:Operating a large distributed system in a reliable way: p... by learnfromstory on Wednesday July 17, 2019 @09:21PM (#20465762)
Which is a good reason to hedge and replicate but NOT a reason to alert on high CPU usage of single computers.
[ Parent | on HN ]
Re:Operating a large distributed system in a reliable way: p... by packetslave on Wednesday July 17, 2019 @10:42PM (#20466229)
You definitely want to TRACK cpu usage on individual hosts, but, yeah, I would alert on service latency instead. Symptom, not cause.
[ Parent | on HN ]