News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by andix on Monday November 16, 2026 @08:01PM (#50526000)
External load can be blocked/limited extremely easily for an internet service. Just block all incoming connections or limit ip ranges at firewall level. Load can be incrementally increased by unblocking more and more ip ranges.

(It might be better not to use CIDR ranges, but use the lower bits instead if possible. For example starting to unblock all IPs that end with .80 until .90, to get more random geographical distribution)

Which leads to my suspicion it was internal load that brought the system down. Communication between the servers (like delivering queued up messages from one O365 account to another), and not external access from users and apps.

Re:Microsoft blames monster email outage on software defect ... by dboreham on Monday November 16, 2026 @08:57PM (#50526426)
Quick note that sometimes it's not that simple. I've worked on a somewhat similar system (although overall much smaller in scope). What we found was a) clients hop between networks and hence IP addresses quite frequently: mobile devices. And b) with sync type protocols you can't just randomly serve whatever sliced and diced subset of the incoming traffic you like. What happens is that many devices end up never achieving current sync state, so they re-try. In our service we had to implement a protocol-aware, and device ID-aware filtering scheme. It had the property you mention where you could dial the proportion of load up or down, but it did it across the known population of devices: if a device got white-listed at t=0 it remained white listed for the remainder of the day. Once it achieved up to date sync it got put into another load bucket. The bucket of devices transitioning from out of sync to synced up was what we limited. It was only after we implemented such a scheme that we got past our thundering heard / can't bring the service back up after an outage problems.
Re:Microsoft blames monster email outage on software defect ... by andix on Monday November 16, 2026 @09:29PM (#50526655)
I don't get this issue. The vast majority of clients would not change IP during a short sync that takes maybe a few seconds. And a switchover from an unblocked to a blocked IP would either result in a partial or in a failed sync, without any consequences.

If the load balancer already does some connection tracking, it can just keep serving existing connections and only drop new ones. If connection tracking is not available, the IP blocklist would be a suitable option.

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @03:31AM (#50528907)
Clients that wake up and sync can get different IPs on every sync on some networks because of carrier NAT.

IP blocking is generally not what you want though.

Re:Microsoft blames monster email outage on software defect ... by andix on Tuesday November 17, 2026 @08:45AM (#50531288)
> IPs on every sync on some networks because of carrier NAT

Okay, they get different IPs during the day, fine. But it's still kind of randomized. Stepping through IP ranges will reduce load and give access to more people.

I really don't get the issue you are seeing with this.

Re:Microsoft blames monster email outage on software defect ... by psd1 on Tuesday November 17, 2026 @08:55AM (#50531418)
IP blocking is dirt cheap on your edge routers. Application-aware binning needs at least your app gateway to have spare capacity. I can see that going tits-up.
Re:Microsoft blames monster email outage on software defect ... by tremon on Tuesday November 17, 2026 @07:37AM (#50530583)
> Which leads to my suspicion it was internal load that brought the system down

IIRC Microsoft posted a comment around 10pm where they discussed rate-limiting external connections, and many reactions on last week's HN thread were like "wait, you so didn't even drop load before now?" So internal load may have been a contributing factor, but it's by no means certain given that they didn't stop serving clients.