News for hackers. Stuff that matters.
Re:Australian employees now have the right to ignore work em... by potamic on Monday August 26, 2024 @05:07AM (#41355280)
What metrics do you alert on? How do you distinguish between error due to faulty database client vs error due to database disk failure?
Re:Australian employees now have the right to ignore work em... by majewsky on Monday August 26, 2024 @05:48AM (#41355519)
Taking my managed container image registry service as an example.

- The only critical alert that can actually page people is if the blackbox test fails. Every 30 seconds, it downloads a test image and if the contents don't match the expectation, an alert is raised (with some delay).

- Warning alerts are mostly for any errors being returned from background tasks, but these are only monitored during business hours.

Re:Australian employees now have the right to ignore work em... by perfect_wave on Monday August 26, 2024 @10:39AM (#41357589)
i dont see how that is separated from the underlying infra. If the network/server/some dependency goes down, the blackbox test will fail and you'll get paged.
Re:Australian employees now have the right to ignore work em... by silisili on Monday August 26, 2024 @03:04PM (#41360554)
You can test for this. For example, we had routines that were called on repeated HTTP failures that would then get 5 or so of the top US websites. If those fail too, it moves from an application error to an infra one.
Re:Australian employees now have the right to ignore work em... by dullcrisp on Monday August 26, 2024 @05:30AM (#41355407)
Define SLOs based on what can realistically be achieved with underlying infrastructure, only alert if those SLOs are breached?
Re:Australian employees now have the right to ignore work em... by sgarland on Monday August 26, 2024 @06:42AM (#41355841)
If your endpoint is failing, it might be you. If everyone’s endpoint is failing, it’s almost certainly not you.