Re:Metastable Failures in Distributed Systems [pdf]
by hinkley on Monday October 04, 2021 @08:52PM
(#28754287)
What bothers me most about The Cloud is that in every other era of networked computing, the people who forgot the https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... would get smacked in the face with it in a matter of months or a few years at most. But now, people seem to be getting away with it, and so nobody is learning.
I suspect 'the problem' is in several parts.
One, there is one administrator, after a sort, and on top of that the amount of real-time monitoring is way up from the Sun era, and so humans can and do intervene more often. Less interactive systems fail for longer before anyone can rectify the situation. The network still isn't reliable, but the intervals are short and it's difficult to build a narrative about how much of a 'failure' it is for a service to be down for 10 minutes versus 8 hours.
Two, operating systems are still struggling to saturate high end networking hardware from a single process, let alone a single thread in that process. If you send data as fast as you can, you can still be doing the same from a couple other processes, so it takes longer to notice the bottleneck. Noisy neighbors can't completely shout you down.
The ones I'm still scratching my head over are homogenous networks and static topologies. Those really are affecting us, but probably get obscured in the statistics to a degree where they can be ignored by most of us, or at least can't be converted into an action item concretely enough, which is effectively the same outcome. They hide in response variability, among other places.
I suspect 'the problem' is in several parts.
One, there is one administrator, after a sort, and on top of that the amount of real-time monitoring is way up from the Sun era, and so humans can and do intervene more often. Less interactive systems fail for longer before anyone can rectify the situation. The network still isn't reliable, but the intervals are short and it's difficult to build a narrative about how much of a 'failure' it is for a service to be down for 10 minutes versus 8 hours.
Two, operating systems are still struggling to saturate high end networking hardware from a single process, let alone a single thread in that process. If you send data as fast as you can, you can still be doing the same from a couple other processes, so it takes longer to notice the bottleneck. Noisy neighbors can't completely shout you down.
The ones I'm still scratching my head over are homogenous networks and static topologies. Those really are affecting us, but probably get obscured in the statistics to a degree where they can be ignored by most of us, or at least can't be converted into an action item concretely enough, which is effectively the same outcome. They hide in response variability, among other places.