News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @09:53AM (#50518217)
Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.

But if low is 64 and high is a whole cluster, now that's a different kind of maths.

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:18AM (#50518509)
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
Re:Microsoft blames monster email outage on software defect ... by jakewins on Monday November 16, 2026 @04:28PM (#50523546)
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.

We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.

This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.

Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.

Re:Microsoft blames monster email outage on software defect ... by awesome_dude on Monday November 16, 2026 @06:20PM (#50524996)
When i saw the lock contention - I immediately thought distributed lock component
Re:Microsoft blames monster email outage on software defect ... by aspbee555 on Monday November 16, 2026 @07:14PM (#50525549)
I have found that AI really loves using locking/Mutex when working with rust and I have to specifically tell it to not use any locking/use atomics/swaps instead
Re:Microsoft blames monster email outage on software defect ... by nl on Monday November 16, 2026 @11:11PM (#50527331)
> So, it's more the former. The testing machine is probably a 8 year old server on an office shelf

I doubt this very much.

The load testing env is the Office 365 load testing environment. I'd be surprised if it is less than 10,000 cores, and the machines are probably good but older; possibly 50 core Xeons instead of 128 core EPYC or something.

It seems likely that the test env was setup to mirror one of the smaller 365 regions a few years ago and the only updates were when machines were replaced.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @08:36AM (#50531169)
Mutices are hard, concurrency is hard, we don't notice the deficiencies in the Linux stack as much for some reason. Nobody's explained why my GUI is laggy while running a crypto miner (for warmth) at nice level 20.
Re:Microsoft blames monster email outage on software defect ... by malkia on Tuesday November 17, 2026 @12:16PM (#50534705)
I believe this must be NUMA cache line contention. If CI testing involved a VM, the VM was most likely cordoned to run in single VM node, or at least not crossing (because it does not make sense otherwise), unless it was special VM to test this case.

I've seen NUMA cache line contention affecting us for real (on Windows, for our game tools). Limiting certain tools to run locked under one NUMA node helped a lot. It's real.

The updated CoreInfo https://learn.microsoft.com/en-us/sysinternals/downloads/cor... shows this well