News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @05:50AM (#50529808)
> efficiency hates redundancy

We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @05:57AM (#50529849)
Basic intuition: if you handle 900 users per minute with a capacity of 1000, and then freeze for one minute or just get an above average usage burst, it'll take 9 minutes to use that excess capacity of 100 to serve that backlog of 900.

If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.

I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @11:57AM (#50534380)
This is the “common sense intuition isn’t” situation. Most people don’t even get this far, as is evidenced by their surprise at why things take so long to get back to normal.

Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.

Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Tuesday November 17, 2026 @02:44PM (#50537082)
thanks guys
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @09:48PM (#50541237)
Related: if a network link is at 90% utilisation, then on average each packet will be stuck behind 9 other packets, in a simplistic model. There's a 90% chance the link is already occupied when the new packet arrives, an 81% chance that slot and the next one are both already occupied, a 72.9% chance that slot and the next two are already occupied, etc. the average delay is 1/(1-r) which is 10 at 90%, 100 at 99%, etc. this works for any type of queue, such as a line to buy movie tickets.
Re:Microsoft blames monster email outage on software defect ... by Choad_20261104 on Tuesday November 17, 2026 @02:49PM (#50537145)
school probably does teach it, students likely don't absorb it. Mine came in a Fluid Dynamics course believe it or not.

Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.

Re:Microsoft blames monster email outage on software defect ... by hinkley on Tuesday November 17, 2026 @07:19PM (#50540196)
Mine didn't, but I got a few things I expected everyone else to learn that nobody has, like distributed computing.

The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.