Re:Giving a Shit as a Service
by 8n4vidtmkvmk on Sunday October 29, 2023 @12:17PM
(#38059985)
I didn't mind staying late some days to see whether or not my migration would blow up production or not. But oncall suuuuuçks.
Cleaning up my own mess is one thing, cleaning up some one else's nonsense is another.
Oncall done right will involve a feedback mechanism to the originators of the root cause.
Things happen all the time.
So lending a hand with emergencies that even could have been avoided is ok to help someone new to it.
Caring via software or infrastructure for users is often about effort as you have described.
At my last job, there was an incident (thankfully discovered in working hours) caused by a supplier accidentally deleting the country of Austria from some crucial database. You can't anticipate everything like that, but the show must go on ...
With that said, there are some types of software that exist to deal with complexity external to itself. External systems (integrations, screen scraping systems, etc.) can break without fault internally, but need to be fixed ASAP.
One might say, "You built it poorly by deciding to build this at all." And sometimes that's true. But I've been in plenty of situations where the options were 1) don't solve the customer problem, don't pass go, don't collect $200. Or, 2) build finicky software that works but needs to be babied... and hire engineers that don't mind doing this type of work.
That last bit is critical. Some engineers like the heroics and drama of it. Some people can't help but run headfirst into the fire... and there's jobs for them there :-)