[Disclaimer: I work for Microsoft]

I'm super impressed that they responded so quickly. When a first notification comes in, it is rarely with a fully fleshed out diagnosis. It is usually some runner/monitoring tool sending you an alert saying something is busted. If you get lucky, you had enough monitoring tools across the system to atleast get close to the source. If not, you have some debugging ahead of you. And then, you need to figure out what a good fix is and given that the entire system is in a wobbly state, a bad fix could make the situation worse. And then, you figure out how to rollout the fix and actually make the change. If you're smart, you'll do it in a staged manner and be able to roll back the moment something goes wrong.

In short, these things take time. Going from notification to working fix in 90 minutes for what sounds like a nasty network hardware issue is very good.

Agreed. 100 minutes from first detection through diagnosis to solution is an excellent turnaround time, and although others have mocked them for plugging their architecture in the post, it's a credit to their architecture that the simplest solution -- flood the system with additional capacity -- was available and so quick to implement.

[um, I work for Yahoo, but do you really need a disclaimer on a compliment?]

I try to mention that whenever a major company which competes with Microsoft in some way is involved. Even the most innocent of statements have a way of getting badly misinterpreted, taken out of context, etc.

I just re-read the article and I was mistaken. I thought they determined it was a capacity problem immediately, but it sounds like that's actually what took so long to figure out.

I work for a web company, and a 100 minute outage is considered absolutely outrageous, but of course it happens.

It sounds like nothing to do with network hardware, they're using 'router' to describe their software that directs gmail HTTP requests around.