Automattic always sounds awesome to work for - I know a couple of people who work there as well.

In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the opposite approach of asking "are you OK?" every 2 minutes. The only solution I've found is to all sit on a call, and pay attention to people's tone of voice, but I don't know whether that actually contributes to people's stress levels.

One of the greatest values of good technical manager is to run interference/shield for your team. Establish a chain of communication, or if there isn't one in place BE the active reporter, to ensure your hands-on-keyboard people aren't interrupted every 5 minutes (and this is a realistic timeline) for updates from various managers, execs, C-levels, and so on.

Otherwise, it's mostly things you can do before and after an incident like this.

Depending on your role and place in hierarchy, yes DO sit on calls, even if you are quiet; to understand what's going on, to help prioritize if needed, to see if your experienced "spidey sense" catches some risk others may not even though they're closer to the problem (or precisely because of it), encourage, guide, and focus; but all of that without being overbearing or making them feel watched. You need to feel like an ally, part of the team, a friend as opposed to political officer. This again is best achieved ahead of time by demonstrating your value to the team by being supportive and effective always.

The honest truth - if you're an engineering manager and you're no longer anywhere near the code then just at the heat of the moment there really isn't much you can do, aside from what you've been doing: listening on the call and doing a minimal check-in.

The main thing you can do is after all is over to make people feel awesome, give them recognition in all the ways possible: bonus, extra vacation days to compensate for the crunch time, etc

But if you still have your technical chops - jump in the trenches and debug!

Best thing managers can do in these situations is deal with internal politics - be the gateway for information going higher up the chair. The last thing the actual engineering team needs is direct calls from C-level execs.

There's avid communication and coordination is all done via text (Slack/IRC). As long as you're following along, I think you can spot it pretty well. People are also more used to managing their own stress and time. They will let the people know if they need to step out for a few minutes so the workload could be shifted/paused to give them time to step away.

This is a benefit of a global company, not working remote. You could do all that stuff in multiple offices around the world.

In a global but not-remote company the team responsible for the particular service that failed is probably still concentrated in one office. Bringing in people from other offices who aren't familiar with the problem service probably isn't that helpful.

This has nothing to do with remote.

Remote teams are usually concentrated in the same timezone too. In-office global companies make a point of having at least a few teams owning a service for this very reason.

You can have knowledge silos in either kind of company. We didn't magically get better at spreading work around or writing documentation just because we went remote.

From my experience there are a few engineers that are usually key to fixing such issues (not always the same people, depending on the failure). Handing-off work is more for day to day work like handeling a support ticket. So the key engineers need to work crazy hours when things like that happen. But that's OK, ideally these things are rare and people need to work crazy hours only in those rare situations. I'm super protective about my evening/weekend time but I wouldn't have any problem working the entire weekend like crazy when something like this happens. If it happens too often I will consider finding a better place to work.

At Automattic, we handle this by offering a 3-month sabbatical every 5 years and a “minimum” 20 day a year vacation policy. The idea being that no one person should hold knowledge to key systems for too much long.