Re:Twitter alerted employees all office buildings closed and...
by gjulianm on Friday November 18, 2022 @02:35AM
(#33651520)
That’s very naive. We’re talking about a distributed system with thousands of servers, with code from people who might not even work there anymore. Yeah, maybe if a server gets a high disk usage alert because of runaway logs you might fix it… but maybe that fix causes another problem, or maybe the alert is in a system that you don’t even know its purpose. Maybe 72h is just what it takes to you understand the internal tooling and how to do any action there.
[ Parent
| on HN ]
Re:Twitter alerted employees all office buildings closed and...
by LambdaComplex on Friday November 18, 2022 @03:02AM
(#33651742)
> code from people who might not even work there anymore
I think that's most of the code, at this point
Re:Twitter alerted employees all office buildings closed and...
by AYBABTME on Friday November 18, 2022 @03:40AM
(#33651978)
Basic firefighting often revolves around the same set of problems. It's not that naive. Tell me I'm wrong? I work with systems like this. Realistically most problems that show up by themselves (ie not caused by bad code) often have very common mitigations/remedies. There's surely some super complex interaction going on but bugs showing up of this complex nature are usually due to code changes, or scaling events that reach a tipping point. Odds are those aren't the bread and butter of problems faced by Twitter _right now_.
Re:Twitter alerted employees all office buildings closed and...
by gjulianm on Friday November 18, 2022 @04:12AM
(#33652217)
> Realistically most problems that show up by themselves (ie not caused by bad code)
Well that goes the first assumption.
> or scaling events that reach a tipping point
Such as having applications running without restarts for a long time due to deploy freezes?
> Odds are those aren't the bread and butter of problems faced by Twitter _right now_.
Right now, exactly this moment, probably not. But if those were the only worrying events that could happen in the short term, I doubt anyone would need to offer any help to Twitter.
> due to code changes
Well, Musk has said that he wants to do "hardcore engineering" implying there will be a lot of changes coming up. That surely can't go wrong.
Re:Twitter alerted employees all office buildings closed and...
by brohee on Friday November 18, 2022 @10:22AM
(#33655987)
The World Cup is starting Sunday. That usually would be an all hands on decks event at least ops wise, but there are very few hands left and the door to the deck is closed anyway...
Re:Twitter alerted employees all office buildings closed and...
by ilyt on Friday November 18, 2022 @06:39AM
(#33653261)
In most cases that's enough to keep it running, even if you need to learn 100x of that to start making changes.
But if infrastructure wasn't managed like shit it will probably just keep on chugging