Why Twitter didn’t go down: From a real Twitter SRE

Re:Why Twitter didn’t go down: From a real Twitter SRE by j3th9n on Tuesday November 22, 2022 @02:38AM (#33703123)

The OOM killer doesn't restart (randomly, unless configured) killed processes, it just kills.

[ Parent | on HN ]

Re:Why Twitter didn’t go down: From a real Twitter SRE by pkolaczk on Tuesday November 22, 2022 @02:56AM (#33703209)

If it's deployed in K8s, it would be restarted automatically after dying.

[ Parent | on HN ]

Re:Why Twitter didn’t go down: From a real Twitter SRE by TylerE on Tuesday November 22, 2022 @03:04AM (#33703252)

Then you have two problems.

[ Parent | on HN ]

Re:Why Twitter didn’t go down: From a real Twitter SRE by ce4 on Tuesday November 22, 2022 @08:10AM (#33705344)

Unless the OOM-killer kills the wrong process. Ages ago we had a userspace filesystem (gpfs) that was of course one of the oldest processes around and it consumed lots of RAM. When the oom killer started looking for a target, of course one of the mmfsd processes was selected and it resulted in instantaneous machine lockup (any access to that filesystem would be blocked forever in the system call which depended on the userspace daemon to return, alas never returning). Was funny to debug

[ Parent | on HN ]

Re:Why Twitter didn’t go down: From a real Twitter SRE by j3th9n on Tuesday November 22, 2022 @09:43AM (#33706303)

You can prevent a process from being killed by OOM killer: https://backdrift.org/oom-killer-how-to-create-oom-exclusion...

[ Parent | on HN ]