News for hackers. Stuff that matters.
Re:Facebook Is Not Working by mrweasel on Tuesday March 05, 2024 @11:02AM (#39605297)
One weekend I was on-call, we had a system we ran for the government, but authentication was handled by another government owned service, hosted and managed by another company.

A user called in early Saturday evening, saying that the system was down. After a bit of debugging, wondering where our alerting had failed, I concluded that the system was perfectly fine, but the authentication service had been returning 500 errors for around an hour. When I called the user back he made a comment that rather changed how I think about monitoring and systems. He says: Well, from my point of view it really doesn't matter which part isn't working, it's just down.

Re:Facebook Is Not Working by muttled on Wednesday March 06, 2024 @12:50PM (#39618639)
I've learned the same lesson, and why things like ping checks aren't enough. Even just a WGET of the front page isn't enough. You need the monitor to login and replicate at least part of the user experience if you have a complex setup. It's very embarrassing when someone tells a user "shows it's up from my side." and they're relying on a naïve monitor.