This is good advice imho.

Away from things that can be automated and are important you have a checklist and tick things off with a pen. Add $this to the list.

Atul Gawande is worth reading in general and on this topic. [1] He turned it into a book I haven't yet read.

[1] https://www.newyorker.com/magazine/2007/12/10/the-checklist

What is the best way to set up those alerts? Specifically how do you set something up that knows if unexpected restarts happened? Is there a dbus or similar event you can listen for?

Worth looking into prometheus. in basic form that'd be gathering metrics from https://github.com/prometheus-community/systemd_exporter on your hosts, and configure alerting in grafana or prometheus alertmanager to notify when a threshold is exceeded

> The reasonable way to notice is to have alerts for any unexpected restarts.

Yes, and to have alerts, I have resorted to write my own small tools [0][1] at the end of the day. Railgun proved to be very useful, but for smaller things, I'm writing the second one.

[0]: https://git.sr.ht/~bayindirh/railgun

[1]: https://sr.ht/~bayindirh/nudge/

In a kubernetes environment, systemd auto-restarting a process inside of a container can hide problems like the article says - if it successfully restarts the process before a liveness probe can pick it up, even if you monitor something like container restarts, you could easily miss this.

Are you talking about running systemd inside a container? Feels like an unnecessary layer.

> The reasonable way to notice is to have alerts for any unexpected restarts. Relying on noticing intermittent service disruption is bound to fail.

I'd argue that unexpected restarts should alert beyond a threshold. Alerting on every occurrence is too noisy. If an individual unit failure causes a service disruption architecture improvements are needed.

Don't you want to know if something restarts unexpectedly? It's a bug that should be understood and fixed. (If it's not a bug then it's not unexpected.)

But if you have too many alerts, you have to remember to check those, and the temptation is to skim over them, and then you'll inevitably miss stuff. Then you need an alert system to alert you to the really important alerts.