Re:Latency Sneaks Up on You
by MatteoFrigo on Friday August 27, 2021 @10:01AM
(#28327895)
Great article, and a line of reasoning that ought to be more widely known. There is a similar tradeoff between latency and utilization in hash tables, for essentially the same reason.
The phenomenon described by the author can lead to interesting social dynamics over time. The initial designer of a system understands the latency/utilization tradeoff and dimensions the system to be underutilized so as to meet latency goals. Then the system is launched and successful, so people start questioning the low utilization, and apply pressure to increase utilization in order to reduce costs. Invariably latency goes up, customers complain. Customers escalate, and projects are started to reduce latency. People screw around at the margin changing number of threads etc, but the fundamental tradeoff cannot be avoided. Nobody is happy in the end. (Been through this cycle a few times already.)
Re:Latency Sneaks Up on You
by dharmab on Friday August 27, 2021 @06:35PM
(#28333692)
In my org, we define latency targets early (based on our user's needs where possible) and then our goal is to maximize utilization within those constraints.
The phenomenon described by the author can lead to interesting social dynamics over time. The initial designer of a system understands the latency/utilization tradeoff and dimensions the system to be underutilized so as to meet latency goals. Then the system is launched and successful, so people start questioning the low utilization, and apply pressure to increase utilization in order to reduce costs. Invariably latency goes up, customers complain. Customers escalate, and projects are started to reduce latency. People screw around at the margin changing number of threads etc, but the fundamental tradeoff cannot be avoided. Nobody is happy in the end. (Been through this cycle a few times already.)