News for hackers. Stuff that matters.
Re:Facebook-owned sites were down by tristor on Monday October 04, 2021 @02:44PM (#28750583)
> It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck.

As someone who formerly did Ops for many many years... this is not accurate. Even in a well organized company there are usually stakeholders at every level on IM calls so that they don't need to play "telephone" for status. For an incident of this size, it wouldn't be unusual to have C-level executives on the call.

While those managers are mostly just quietly listening in on mute if they know what's good (e.g. don't distract the people doing the work to fix your problem), their mere presence can make the entire situation more tense and stressful for the person banging keyboards. If they decide to be chatty or belligerent, it makes everything 100x worse.

I don't envy the SREs at Facebook today. Godspeed fellow Ops homies.

Re:Facebook-owned sites were down by Salgat on Monday October 04, 2021 @03:30PM (#28751020)
I think it comes down to the comfort level of the worker. I remember when our production environment went down. The CTO was sitting with me just watching and I had no problem with it since he was completely supportive, wasn't trying to hurry me, just wanted to see how the process of fixing it worked. We knew it wasn't any specific person's fault, so no one had to feel the heat from the situation beyond just doing a decent job getting it back up.
Re:Facebook-owned sites were down by LordHumungous on Monday October 04, 2021 @06:45PM (#28753265)
C levels don't sit on the call with engineers. They aren't that dumb. Managers will communicate upward.
Re:Facebook-owned sites were down by tristor on Tuesday October 05, 2021 @12:44AM (#28755578)
That greatly depends on the incident and the organization. I’ve personally been on numerous incident calls with C-level folks involved.
Re:Facebook-owned sites were down by ikiris on Tuesday October 05, 2021 @01:30AM (#28755783)
Yeah hell, I've ended up with one of the big names as my comm's lead.

That in itself was stressful, and became an example case later.