Re:Don't trust default timeouts
by user5994461 on Saturday August 29, 2020 @05:57PM
(#24319707)
That number sounds like really bad advise to me. Should be more like 99.99% in my experience.
Internal services have extremely low response time during normal operation (p99 around a second) but then the database will start a snapshot or a large analytics query hits on the week end (high IO) and the latency is through the roof for a short while. Too bad if services have short timeouts, they're all failing all requests now for no reason.
p99 is normal operation. Services shouldn't be configured to systematically fail for 1% of operations.
Re:Don't trust default timeouts
by hderms on Tuesday September 01, 2020 @02:37PM
(#24345193)
fair enough, that's why they call out that you need to load test it and actually determine that the value you set meets expectations. Agreed that blindly setting a value is problematic
Internal services have extremely low response time during normal operation (p99 around a second) but then the database will start a snapshot or a large analytics query hits on the week end (high IO) and the latency is through the roof for a short while. Too bad if services have short timeouts, they're all failing all requests now for no reason.
p99 is normal operation. Services shouldn't be configured to systematically fail for 1% of operations.