News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @10:36AM (#50518712)
Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.

So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:45AM (#50518835)
Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.

This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.

Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @10:46AM (#50518859)
Sounds reasonable.

I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:49AM (#50518906)
IP is often hard on the proxy/load balancer, you get a really large list, which becomes expensive.
Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @10:51AM (#50518920)
This is also what I thought, so why I asked, actually.
Re:Microsoft blames monster email outage on software defect ... by toast0 on Monday November 16, 2026 @07:11PM (#50525515)
Selectively allowing IPs is a reasonable way to do it. If you think about it, it's terrible, but it will get you going, and it's pretty easy do something like start with allow nothing, then allow all ips under 2.0.0.0, then 3.0.0.0, etc.

Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.

Re:Microsoft blames monster email outage on software defect ... by DenisM on Tuesday November 17, 2026 @01:04PM (#50535518)
I’m wondering if you’re preparing for cold start ahead of time you could create a better cold start map that’s distributed more evenly for more predictable turn up, and also with better affinity so you can you report progress in a way people can understand ( eg state by state). It would be a pain to load and activate the map into edge routers in a hurry, but maybe be necessary. Some superscalers are deep into writing their own networking firmware, this could go into design requirements for text revision.

Another idea is to have different clients use different ports (enterprise customers on one port, SMB on another) and turn up in the order of SLA.

Re:Microsoft blames monster email outage on software defect ... by nijave on Monday November 16, 2026 @08:21PM (#50526176)
They could drop traffic at L3 before it even hits their servers.
Re:Microsoft blames monster email outage on software defect ... by lokar on Tuesday November 17, 2026 @11:07AM (#50533387)
That's what we are talking about. But you need a way to manage that L3 traffic by client.