Re:COBOL underpins the entire financial system
by rwmj on Sunday November 22, 2020 @10:04AM
(#25177953)
If it's Linux there are various ways to find out who is sending that signal, killsnoop being one (part of bcc: https://github.com/iovisor/bcc), another using a trivial systemtap script (https://www.ibm.com/support/pages/systemtap-kill-who-killed-...).
Re:COBOL underpins the entire financial system
by yourapostasy on Sunday November 22, 2020 @12:36PM
(#25179261)
I <3 Linux for just this kind of community. Thank you, you're righteous!
They let this go on for about six years before I arrived and saw it. It took me pointing out that there is a decent chance this is caused by the process relying upon something in the OS kernel that isn't quite aligned to the kernel documentation but is good enough, and the immediate respawn behavior might rely upon that in turn. If the kernel ever is "fixed", then this audit-compliance-related process suddenly stops and doesn't respawn, or worse, won't start at all, becoming top of mind with all the senior management. It is always cheaper to fix problems in the small before they become problems no one can ignore.
The urge to sweep problems under the rug and move on is very powerful within our industry. Until you've done the same yourself and been bitten so many times, your scar tissue twitches every time you see the same pattern again. These days, I treat code problems like I treat cleaning up messes while cooking: I clean up as I go along. My scar tissue thanks me now. There is a delicate balance between addressing these problems in the small and bikeshedding, though.
BTW, I performed an strace that revealed a SIGKILL just pops in without anyone or any known process issuing it. The application developers suspect something in the OS, so we're now engaged with the application support team, OS support team, and our own internal OS support team to trace it down and beat it into submission.