Drgn: How the Linux kernel team at Facebook debugs the kernel at scale

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by silisili on Tuesday December 14, 2021 @02:19AM (#29549125)

Different mindsets. I was cutting my teeth on a set of 12 static servers/processes, with no autoscaling or auto reboots. Memory leaks were rare but reported. Usually FD leaks were culprits. So learned a ton about how to write really correct applications, and today it just feels like none of that matters. And in complete fairness, with today's ops tools, it probably doesn't.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by Arainach on Tuesday December 14, 2021 @02:26AM (#29549165)

At scale, server failures are going to happen. Maybe it's a power outage. Maybe it's a failed drive. Maybe it's a kernel crash. Only one of those can be remedied (not even prevented) with careful debugger investigation, so you're going to have to do the work to be able to handle such failures gracefully. At that point, unless the leaks are causing outages so rapidly that the machines can't handle, then rebooting becomes not that big of a deal, while wasting senior eng days trying to track them down is a real cost.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by hvgk on Tuesday December 14, 2021 @03:04AM (#29549371)

There are some nasty escalation scenarios I’ve seen in that space. Cost increases as does workload backlogs and latency.

Eventually something breaks and causes an outage.

All leaks should be resolved lest you end up with floods.

[ Parent | on HN ]