News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by ncr100 on Monday November 16, 2026 @09:29AM (#50517969)
They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.

Is this commonly acceptable? I'd imagine it's a costly trade-off?

Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @09:53AM (#50518217)
Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.

But if low is 64 and high is a whole cluster, now that's a different kind of maths.

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:18AM (#50518509)
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
Re:Microsoft blames monster email outage on software defect ... by jakewins on Monday November 16, 2026 @04:28PM (#50523546)
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.

We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.

This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.

Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.

Re:Microsoft blames monster email outage on software defect ... by awesome_dude on Monday November 16, 2026 @06:20PM (#50524996)
When i saw the lock contention - I immediately thought distributed lock component
Re:Microsoft blames monster email outage on software defect ... by aspbee555 on Monday November 16, 2026 @07:14PM (#50525549)
I have found that AI really loves using locking/Mutex when working with rust and I have to specifically tell it to not use any locking/use atomics/swaps instead
Re:Microsoft blames monster email outage on software defect ... by nl on Monday November 16, 2026 @11:11PM (#50527331)
> So, it's more the former. The testing machine is probably a 8 year old server on an office shelf

I doubt this very much.

The load testing env is the Office 365 load testing environment. I'd be surprised if it is less than 10,000 cores, and the machines are probably good but older; possibly 50 core Xeons instead of 128 core EPYC or something.

It seems likely that the test env was setup to mirror one of the smaller 365 regions a few years ago and the only updates were when machines were replaced.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @08:36AM (#50531169)
Mutices are hard, concurrency is hard, we don't notice the deficiencies in the Linux stack as much for some reason. Nobody's explained why my GUI is laggy while running a crypto miner (for warmth) at nice level 20.
Re:Microsoft blames monster email outage on software defect ... by malkia on Tuesday November 17, 2026 @12:16PM (#50534705)
I believe this must be NUMA cache line contention. If CI testing involved a VM, the VM was most likely cordoned to run in single VM node, or at least not crossing (because it does not make sense otherwise), unless it was special VM to test this case.

I've seen NUMA cache line contention affecting us for real (on Windows, for our game tools). Limiting certain tools to run locked under one NUMA node helped a lot. It's real.

The updated CoreInfo https://learn.microsoft.com/en-us/sysinternals/downloads/cor... shows this well

Re:Microsoft blames monster email outage on software defect ... by notyourwork on Monday November 16, 2026 @11:02AM (#50519087)
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Monday November 16, 2026 @11:28AM (#50519438)
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience. some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult

plus, mistakes are the essence of learning and sometimes mistakes are necessary.

likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @05:50AM (#50529808)
> efficiency hates redundancy

We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @05:57AM (#50529849)
Basic intuition: if you handle 900 users per minute with a capacity of 1000, and then freeze for one minute or just get an above average usage burst, it'll take 9 minutes to use that excess capacity of 100 to serve that backlog of 900.

If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.

I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @11:57AM (#50534380)
This is the “common sense intuition isn’t” situation. Most people don’t even get this far, as is evidenced by their surprise at why things take so long to get back to normal.

Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.

Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Tuesday November 17, 2026 @02:44PM (#50537082)
thanks guys
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @09:48PM (#50541237)
Related: if a network link is at 90% utilisation, then on average each packet will be stuck behind 9 other packets, in a simplistic model. There's a 90% chance the link is already occupied when the new packet arrives, an 81% chance that slot and the next one are both already occupied, a 72.9% chance that slot and the next two are already occupied, etc. the average delay is 1/(1-r) which is 10 at 90%, 100 at 99%, etc. this works for any type of queue, such as a line to buy movie tickets.
Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Tuesday November 17, 2026 @02:49PM (#50537145)
school probably does teach it, students likely don't absorb it. Mine came in a Fluid Dynamics course believe it or not.

Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @07:19PM (#50540196)
Mine didn't, but I got a few things I expected everyone else to learn that nobody has, like distributed computing.

The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.

Re:Microsoft blames monster email outage on software defect ... by pwarner on Monday November 16, 2026 @04:51PM (#50523849)
I think MS could afford a bit of a test lab?
Re:Microsoft blames monster email outage on software defect ... by marcta on Monday November 16, 2026 @05:03PM (#50524020)
Yes but something that works fine in a test lab won't necessarily cope when confronted with hundreds of millions of users.
Re:Microsoft blames monster email outage on software defect ... by jamesfinlayson on Monday November 16, 2026 @09:23PM (#50526626)
Yeah, my day job's peak is at the very extreme, thousands of users a second. I believe there is a dedicated set of servers to load test this scenario, which while not cheap, is not cost prohibitive, and it's deemed absolutely critical that the system be able to handle this, so the business is happy to foot the bill.
Re:Microsoft blames monster email outage on software defect ... by Anon1096 on Tuesday November 17, 2026 @05:38AM (#50529721)
Here we're talking far more than a million requests per second. You can't buy the spare hundreds of thousands to millions of servers needed to do a separate non production load test of something at this scale no matter how much money you are willing to spend.
Re:Microsoft blames monster email outage on software defect ... by munchbunny on Monday November 16, 2026 @05:19PM (#50524229)
At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.

Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @05:53AM (#50529824)
Some projects end up slowing rolling restarts down, in the hopes that alarms will start to fire if the new servers are not keeping up like the older version did. Then someone forgets why it’s that way and we are quickly reminded and it gets restored to the old timing.
Re:Microsoft blames monster email outage on software defect ... by awesome_dude on Monday November 16, 2026 @06:18PM (#50524981)
I read that part of their statement as "QA were set a priority to ensure existing hardware wasn't affected, which reduced their (QAs) capacity to test on new hardware which was assumed to be safe"

They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)