News for hackers. Stuff that matters.
Re:The Mystery Blips by encoderer on Saturday November 19, 2022 @11:54AM (#33671832)
Maybe it’s just the trained operator in me but the whole time he’s describing the depressed metrics and blip I’m screaming in my head: it’s exogenous! Check the news! He finally gets there.

For us (at normal company scale) I would actually go look at recent activity, new customers, new workloads we are running, but I guess at a Facebook scale it’s a lot harder to do that.

Re:The Mystery Blips by nerdponx on Saturday November 19, 2022 @11:59AM (#33671906)
One interesting aspect of the story is that the author was a junior at the time (almost brand new to the job, if I read it correctly), and it was one of the more experienced operators who realized it was exogenous.

in addition to being a good story about the site reliability, it's also a great lesson in the value of collaboration, mentorship, and having senior people around whom you can ask for help!

Re:The Mystery Blips by MrAwesome on Sunday November 20, 2022 @01:04PM (#33683431)
Author here, and that's exactly right! I think it was in my second or third month on the job.

And I really agree with your second point too! Without that in-person transfer of tribal knowledge, it would have taken me ages to learn the ropes well enough to take such a high-pressure high-stakes oncall.