The Day “/Proc” Died

Re:The Day “/Proc” Died by colechristensen on Wednesday July 03, 2019 @03:54AM (#20342453)

In ops, I've seen this happen very many times. A linux server running happy and free because the kernel OOM killer murdered the reason that server existed, leaving alone some side process with a memory leak (usually some external service agent or maintenance service run amok). I learned well how to fix that over the years.

(spoiler alert: it was much more often about becoming ornery when devs insisted that their JVM app could have a 12 GB heap on a machine with 12 GB of memory)

[ Parent | on HN ]

Re:The Day “/Proc” Died by mshook on Wednesday July 03, 2019 @12:55PM (#20346153)

That's pretty much why we use vm.panic_on_oom = 1 as you have no idea about what's going to be killed...

[ Parent | on HN ]

Re:The Day “/Proc” Died by colechristensen on Wednesday July 03, 2019 @05:51PM (#20349224)

I am in the camp of having systemd or your service manager of choice restart any killed service or service failing health check and emitting an ERROR or higher log message.

I want my systems to heal themselves. More often than not these memory problems end up being slow leaks which can be effectively permanently resolved with periodic restarts, and the engineering time to fix them is appropriately not prioritized.

I want to know that there has been a problem but I would rather not be forced to do anything about it unless absolutely necessary.

[ Parent | on HN ]