News for hackers. Stuff that matters.

A comment on Running servers and services well is not trivial (2018). Parent comment

Re:Running servers and services well is not trivial (2018) by buffet_overflow on Saturday March 26, 2022 @01:00PM (#30813447)
The problem is when they have more services than sysadmins. While the sysadmins are busy upgrading the git server, the logging infrastructure suffers. They pivot to work on that, now the CI/CD server is down/slow/randomly breaking. But the sysadmin that knew the ins and outs of it left last quarter so the new sysadmins don't want to touch it. Oh, and management doesn't prioritize any of this stuff, so actually jumping two versions on the git server is a much bigger, fragile ordeal now than it was a month ago.
[ Parent | on HN ]
Re:Running servers and services well is not trivial (2018) by CameronNemo on Saturday March 26, 2022 @03:19PM (#30814527)
Exactly.

Our team of 6 sysadmins manages:

- DNS appliances, storage appliances, NTP appliances,

- hypervisors, Dev/stage/prod k8s clusters, some other k8s clusters

- dev/prod Elasticsearch/Logstash/Kibana clusters

- internal GitLab, Jira, Confluence, nautobot, OpenDCIM, a deprecated Twiki

- several internal custom apps

- Probably more I am forgetting.

Nothing gets patched consistently. Everything is neglected to a certain degree.

[ Parent | on HN ]
Re:Running servers and services well is not trivial (2018) by stevekemp on Sunday March 27, 2022 @01:13PM (#30821936)
In a previous job I managed a migration from bitbucket (!) to self-hosted Github Enterprize. As part of the migration I forced the creation of a monthly "backup restore test".

Every month we had a recipe to spin up a new instance of the Github appliance, and import the most recent backup into it.

A lot of systems do tend to get a little neglected over time, and of course different organizations have different priorities, but I insisted on this because I figured for that particular company the Github instance was one of the most critical components - if it is down people can't work.

[ Parent | on HN ]
Re:Running servers and services well is not trivial (2018) by rhizome on Saturday March 26, 2022 @05:54PM (#30815697)
I'm not going to die on this hill, but that seems like a lot of complexity for a company with in-house skills, maybe even the worst of both (in-house vs cloud/managed) worlds.
[ Parent | on HN ]