We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.
If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.
I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.
Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.
Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.
The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.
plus, mistakes are the essence of learning and sometimes mistakes are necessary.
likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful