Re:Signals in Prod: Dangers and Pitfalls
by ojn on Monday October 03, 2022 @11:17PM
(#33076373)
Good writeup, and lots of detail.
But my immediate conclusion isn't "Signals are dangerous and let's go deep on the complexity thereof". Instead, my primary conclusion really is "Someone forgot to write a testcase to make sure log rotation behavior is covered, so of course it might regress".
Re:Signals in Prod: Dangers and Pitfalls
by brian_cloutier on Monday October 03, 2022 @11:41PM
(#33076499)
Both are true.
I would throw in a third conclusion: given that you're directly using signals (without a wrapper to rationalize the interface) your team should be composed of people who fully understand how signals work. Neither the person who wrote this code or the person who reviewed it had internalized the fact that "the default SIGHUP handler kills the process", and therefore using signals was an inappropriate decision for this team.
But this is an exceptionally high bar; most interfaces do not require that your devs have memorized so many details. Signals expose a terrible interface and there is essentially no satisfying way to make use of them. We should expect better from the tools we use.
Re:Signals in Prod: Dangers and Pitfalls
by ok_dad on Tuesday October 04, 2022 @01:26AM
(#33076930)
I dunno, I don't even program in C and when scrolling through the diff I immediately said to myself, "Hey, I bet that callback registration they removed for SIGHUP could be an issue; I would have no-op'ed the callback and left it in there." Of course, I had the privilege of knowing that there was something wrong with that specific code diff relating to signals.
My take is similar to yours: only use tech and features the plurality your team understands. If the people leave that are the only ones who understand the fancy stuff, then you run into this type of issue. At my current job, the guy who setup the database put all kinds of fancy extensions, triggers, and procedures in there and no one knows how to dissect it or what it all does, so now we're running old versions of code in a database container. I am sure someday it will be my job to replace that black box with something else, and probably just move the data from one to the other with ETL methods.
Re:Signals in Prod: Dangers and Pitfalls
by Spivak on Tuesday October 04, 2022 @03:02AM
(#33077354)
If any shop lets their code changes marinate for 24 hours in a test environment then power to you, you have more patience than any developer I've ever worked with. It feels like a uniquely ops-y mentality to roll out changes on the order of days or weeks.
Re:Signals in Prod: Dangers and Pitfalls
by richij on Tuesday October 04, 2022 @06:34AM
(#33078516)
Oh get off my lawn, kid. It's not signals that are dangerous; it's removing code that's dangerous. You thought it was redundant, but it turned out you were wrong.
Re:Signals in Prod: Dangers and Pitfalls
by toast0 on Tuesday October 04, 2022 @01:49AM
(#33077030)
FWIW, if you can get away with it, it's nice to have your log writer check if the file has been rotated when writing if it's been 60 seconds or more since the last check; this allows you to avoid signalling, although you do need to wait to compress the logs, if log compression is part of your rotation.
Re:Signals in Prod: Dangers and Pitfalls
by nightowl_games on Monday October 03, 2022 @10:03PM
(#33076025)
This was a great writeup. I have read a few great low level articles from facebook and it definetly brings up my view of fb. I hope they continue to produce quality content like this.
Re:Signals in Prod: Dangers and Pitfalls
by 0xbadcafebee on Monday October 03, 2022 @11:16PM
(#33076365)
Completely unrelated, but useful: POSIX kill (or kill()) with signal 0 is a portable way to check if a process controllable by the current user is alive.
Re:Signals in Prod: Dangers and Pitfalls
by saagarjha on Tuesday October 04, 2022 @01:45AM
(#33077011)
It doesn’t do anything but returns a result based on the existence of the process and your ability to signal it, which makes it a sort of “dry run” of sorts. Note that such a check is inherently racy, though.
Re:Signals in Prod: Dangers and Pitfalls
by GauntletWizard on Monday October 03, 2022 @10:37PM
(#33076180)
I can't argue with any of this, but like xkcds "Teach wearing a condom"[1], everything points to something having gone horribly wrong to have this lesson.
But my immediate conclusion isn't "Signals are dangerous and let's go deep on the complexity thereof". Instead, my primary conclusion really is "Someone forgot to write a testcase to make sure log rotation behavior is covered, so of course it might regress".
I would throw in a third conclusion: given that you're directly using signals (without a wrapper to rationalize the interface) your team should be composed of people who fully understand how signals work. Neither the person who wrote this code or the person who reviewed it had internalized the fact that "the default SIGHUP handler kills the process", and therefore using signals was an inappropriate decision for this team.
But this is an exceptionally high bar; most interfaces do not require that your devs have memorized so many details. Signals expose a terrible interface and there is essentially no satisfying way to make use of them. We should expect better from the tools we use.
My take is similar to yours: only use tech and features the plurality your team understands. If the people leave that are the only ones who understand the fancy stuff, then you run into this type of issue. At my current job, the guy who setup the database put all kinds of fancy extensions, triggers, and procedures in there and no one knows how to dissect it or what it all does, so now we're running old versions of code in a database container. I am sure someday it will be my job to replace that black box with something else, and probably just move the data from one to the other with ETL methods.
Everyone has a test environment and some people have a prod environment too.
Do you mean in a kind of "time of check vs. time of use" kind of way?
I would suppose that if the answer is negative that would at least stay true no?
[1] https://xkcd.com/463/
This is pretty much inherent in multithreaded environments.