Re:Reflecting on the Soul of a New Machine
by notacoward on Wednesday February 13, 2019 @08:58AM
(#19153083)
OTOH, in the context of a distributed system that has to be upgraded live and the deployment of new code has to be carefully staged across hours or days (e.g. a storage cluster with thousands of machines), such mode switches can be absolutely essential. The process can look something like this.
* Roll out the code with support for a new feature
* Flip the mode switch on a few nodes to use the new feature
* Wait a while, watch for anomalies
* Enable the new feature on more nodes, until it's on all
* Disable support for the old feature (another mode switch)
* Eventually, roll out new code with the old feature entirely removed
It's not quite a whole-system mode switch as in SoaNM, but that's kind of the point. In a distributed system actions do not affect the whole system all at once, so different rules apply.
At one point the CEO injects a critical piece of wisdom by telling the engineers they are not allowed to use a mode switch to preserve backwards compatibility.
In any system, the presence of mode switches often points to where things will go wrong. If a system has two modes, then inevitably, one of them will be more unloved, untested, under documented and unreliable than the other.