News for hackers. Stuff that matters.

anton_

Re:What Toasters and Distributed Systems Might Have in Common on Thursday October 25, 2018 @01:25PM (#18302601)
On non-linearity of software systems. Correct me if I’m wrong but I’m assuming you mean it in a sense of being probabilistic and not easily predictable. Which is an interesting point because a lot of modern algorithms are like that. Let’s take neural networks. Being able to explain what the model is doing and how to make sure it’s correct is a whole area of research. For example, the results of a convolutional net can change drastically by altering a single pixel in an image[1].

We also faced with a problem of explaining a machine learning model that was much simpler than a neural net. It is a global problem and unfortunately I don’t have any solution of the top of my head.

[1] https://arxiv.org/abs/1710.08864

Re:What Toasters and Distributed Systems Might Have in Common on Wednesday October 24, 2018 @07:02PM (#18297219)
Regarding the point that the software systems being more flexible. A nice thing about controllers such as PID is that they deal well with disturbances and changing environments. In this particular case the PID controller runs alongside the system with a fixed sampling rate. It means that it does not really care about dynamics of the system in control. Say we want to update the caching TTL and that affects how the system behaves. The PID algo only cares about the delta of the error over a fixed period of time (sampling rate) and will adjust accordingly. Also, as mirceal mentioned, there are autotuning algos if it gets really bad.

Unfortunately it is a problem if the system changes too much the algo can stop working properly. But that could happen any time you make a major change in any complex system. To me it’s more of a problem of testing and sanity-checking, not of PID or other control algorithms. When you get to a certain point of complexity, testing all possible conditions becomes unrealistic and that might lead to bugs when a big part of the system gets rewritten.

On the point that only a few engineers might know how it works. One of the purpose of the article is actually put an idea out there to make more engineers familiar with the concept and collect some feedback. :)

I’ll update the comment with “On non-linearity of software systems” when I get a chance. This is actually a really good one and I have a few thoughts about it.

Re:What Toasters and Distributed Systems Might Have in Common on Wednesday October 24, 2018 @06:19PM (#18296970)
The author is here. Thank you for a thorough reply, these are both valid points. I have been thinking about them before implementing the algo and this is where my thoughts are at the moment:

PID tuning is a complicated task, but what I found is that the parameter space where the coefficients are "good enough" is quite big. Certainly much bigger than where the coefficients are "perfect". A lot of times changing the coefficients has some effect on the transient state parameters such as rise[1] and settling[2] times, but they still result in a robust controller that drives the value to the setpoint. I agree, this might not be acceptable for some applications where the transient time is very important, but often times even making a P controller does a fairly good job controlling the system. In this case it means a single coefficient that is relatively straightforward to tune.

I’m pretty sure our simulation was not perfect as we scaled down time to speed up the testing cycles, but we didn’t spend too much time on it. It was just a test script to get the initial values of the coefficients. When we got those we could deploy it to staging environment and run a load-testing tool to fine-tune them. Improving the existing system would mean coming up with a way to broadcast the change quickly enough, and that would add more dependencies and synchronization overhead despite being time consuming. Right now everything is done with standalone loosely-coupled services that communicate with each through APIs and cache the results. This is a very good point and I think I’m going to add these details to the post.

[1] https://en.wikipedia.org/wiki/Rise_time [2] https://en.wikipedia.org/wiki/Settling_time