News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by notyourwork on Monday November 16, 2026 @11:02AM (#50519087)
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Monday November 16, 2026 @11:28AM (#50519438)
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience. some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult

plus, mistakes are the essence of learning and sometimes mistakes are necessary.

likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @05:50AM (#50529808)
> efficiency hates redundancy

We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @05:57AM (#50529849)
Basic intuition: if you handle 900 users per minute with a capacity of 1000, and then freeze for one minute or just get an above average usage burst, it'll take 9 minutes to use that excess capacity of 100 to serve that backlog of 900.

If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.

I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @11:57AM (#50534380)
This is the “common sense intuition isn’t” situation. Most people don’t even get this far, as is evidenced by their surprise at why things take so long to get back to normal.

Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.

Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Tuesday November 17, 2026 @02:44PM (#50537082)
thanks guys
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @09:48PM (#50541237)
Related: if a network link is at 90% utilisation, then on average each packet will be stuck behind 9 other packets, in a simplistic model. There's a 90% chance the link is already occupied when the new packet arrives, an 81% chance that slot and the next one are both already occupied, a 72.9% chance that slot and the next two are already occupied, etc. the average delay is 1/(1-r) which is 10 at 90%, 100 at 99%, etc. this works for any type of queue, such as a line to buy movie tickets.
Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Tuesday November 17, 2026 @02:49PM (#50537145)
school probably does teach it, students likely don't absorb it. Mine came in a Fluid Dynamics course believe it or not.

Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @07:19PM (#50540196)
Mine didn't, but I got a few things I expected everyone else to learn that nobody has, like distributed computing.

The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.

Re:Microsoft blames monster email outage on software defect ... by pwarner on Monday November 16, 2026 @04:51PM (#50523849)
I think MS could afford a bit of a test lab?
Re:Microsoft blames monster email outage on software defect ... by marcta on Monday November 16, 2026 @05:03PM (#50524020)
Yes but something that works fine in a test lab won't necessarily cope when confronted with hundreds of millions of users.
Re:Microsoft blames monster email outage on software defect ... by jamesfinlayson on Monday November 16, 2026 @09:23PM (#50526626)
Yeah, my day job's peak is at the very extreme, thousands of users a second. I believe there is a dedicated set of servers to load test this scenario, which while not cheap, is not cost prohibitive, and it's deemed absolutely critical that the system be able to handle this, so the business is happy to foot the bill.
Re:Microsoft blames monster email outage on software defect ... by Anon1096 on Tuesday November 17, 2026 @05:38AM (#50529721)
Here we're talking far more than a million requests per second. You can't buy the spare hundreds of thousands to millions of servers needed to do a separate non production load test of something at this scale no matter how much money you are willing to spend.