But if low is 64 and high is a whole cluster, now that's a different kind of maths.
We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.
This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.
Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.
I doubt this very much.
The load testing env is the Office 365 load testing environment. I'd be surprised if it is less than 10,000 cores, and the machines are probably good but older; possibly 50 core Xeons instead of 128 core EPYC or something.
It seems likely that the test env was setup to mirror one of the smaller 365 regions a few years ago and the only updates were when machines were replaced.
I've seen NUMA cache line contention affecting us for real (on Windows, for our game tools). Limiting certain tools to run locked under one NUMA node helped a lot. It's real.
The updated CoreInfo https://learn.microsoft.com/en-us/sysinternals/downloads/cor... shows this well
plus, mistakes are the essence of learning and sometimes mistakes are necessary.
likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful
We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.
If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.
I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.
Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.
Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.
The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.
Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.
They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)
Is this commonly acceptable? I'd imagine it's a costly trade-off?