News for hackers. Stuff that matters.
Re:Australian employees now have the right to ignore work em... by silisili on Monday August 26, 2024 @03:45AM (#41354879)
That's the benefit of working on a small team/startup. Whoever smelt it dealt it kind of situation. Basically, at first at least, whoever's application is erroring is getting a call when it breaks.

It sounds bad, but it encourages everyone to write more fault tolerant code. Way moreso then a random bigcorp with an on call team.

Re:Australian employees now have the right to ignore work em... by grecy on Monday August 26, 2024 @04:00AM (#41354957)
I used to get called all the time because our in house infrastructure would fall over, and my apps would crash. It didn’t matter how many times I explained my apps couldn’t run without good infra, and that I wasn’t on that team, and that I had no access or authority to do anything… when my apps when down I got called.

So actually, I really don’t like your idea.

Re:Australian employees now have the right to ignore work em... by majewsky on Monday August 26, 2024 @04:08AM (#41354996)
Or, to be more specific, you don't like your company's implementation of their idea.

We have the same setup in my org, but we get to define alerts ourselves. All our own alerts are built so that they don't go off if the underlying infra is borked, and only if there's something we can actually do on our level. We are being kept honest because there is a big kerfuffle when an incident is reported by customers first (instead of alerting).

Re:Australian employees now have the right to ignore work em... by potamic on Monday August 26, 2024 @05:07AM (#41355280)
What metrics do you alert on? How do you distinguish between error due to faulty database client vs error due to database disk failure?
Re:Australian employees now have the right to ignore work em... by majewsky on Monday August 26, 2024 @05:48AM (#41355519)
Taking my managed container image registry service as an example.

- The only critical alert that can actually page people is if the blackbox test fails. Every 30 seconds, it downloads a test image and if the contents don't match the expectation, an alert is raised (with some delay).

- Warning alerts are mostly for any errors being returned from background tasks, but these are only monitored during business hours.

Re:Australian employees now have the right to ignore work em... by perfect_wave on Monday August 26, 2024 @10:39AM (#41357589)
i dont see how that is separated from the underlying infra. If the network/server/some dependency goes down, the blackbox test will fail and you'll get paged.
Re:Australian employees now have the right to ignore work em... by silisili on Monday August 26, 2024 @03:04PM (#41360554)
You can test for this. For example, we had routines that were called on repeated HTTP failures that would then get 5 or so of the top US websites. If those fail too, it moves from an application error to an infra one.
Re:Australian employees now have the right to ignore work em... by dullcrisp on Monday August 26, 2024 @05:30AM (#41355407)
Define SLOs based on what can realistically be achieved with underlying infrastructure, only alert if those SLOs are breached?
Re:Australian employees now have the right to ignore work em... by sgarland on Monday August 26, 2024 @06:42AM (#41355841)
If your endpoint is failing, it might be you. If everyone’s endpoint is failing, it’s almost certainly not you.
Re:Australian employees now have the right to ignore work em... by latexr on Monday August 26, 2024 @07:38AM (#41356173)
Pretty sure your parent poster meant a small overall team. As in, the company is small enough that everyone knows who everyone else is and there’s little to no bureaucracy to reach the right person.

Doesn’t seem like your case at all.

Re:Australian employees now have the right to ignore work em... by watwut on Monday August 26, 2024 @05:22AM (#41355369)
Unless of course the "guilty" is not immediately apparent.

If it happens frequently, the guilty is the process, team lead or whoever runs the things.

Re:Australian employees now have the right to ignore work em... by aqme28 on Monday August 26, 2024 @05:26AM (#41355385)
This is how you end up in situations where no one wants to work on the team that actually needs the most help
Re:Australian employees now have the right to ignore work em... by teeray on Monday August 26, 2024 @08:09AM (#41356361)
> Whoever smelt it dealt it

Until whoever dealt it just leaves the team. Then it’s everyone’s problem.

Re:Australian employees now have the right to ignore work em... by amrocha on Monday August 26, 2024 @08:28AM (#41356478)
I joined a startup a couple years ago, and got handed a poisoned chalice.

It was a project that was critical to the company, but that was not very reliable, and broke overnight very often, sometimes 3-4 days per week.

I was the 4th dev on it. Everyone else who worked on it before had burned out and quit. The dev before me couldn’t tough it out another 2 weeks and quit before i joined.

Eventually I burned out after a year and quit too. All this to say, that’s great when it works, but when it goes bad it’s real bad hahaha.

Re:Australian employees now have the right to ignore work em... by Flop7331 on Monday August 26, 2024 @01:22PM (#41359442)
There's also the call you get when the founder breaks something and blames the person who touched it before them.