Drgn: How the Linux kernel team at Facebook debugs the kernel at scale

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by ajuc on Tuesday December 14, 2021 @02:35AM (#29549209)

I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy.

Depends on the domain.

For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :)

The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by onion2k on Tuesday December 14, 2021 @02:52AM (#29549309)

That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by xorcist on Tuesday December 14, 2021 @03:51AM (#29549621)

It's not that simple. Intermittently rebooting touchscreens make for a bad user experience.

User experience is one of those things that doesn't matter until it suddently does, often when a competitor sells a product that doesn't reboot in the middle of a user operation.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by ajuc on Tuesday December 14, 2021 @03:02AM (#29549365)

If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by onion2k on Tuesday December 14, 2021 @03:43AM (#29549580)

Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by ajuc on Tuesday December 14, 2021 @04:20AM (#29549784)

It still feels dirty though.

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by unwind on Tuesday December 14, 2021 @07:40AM (#29550946)

Wow, what an interesting bug! Was it a case of parsing "1.000" vs "1,000" or something like that?

[ Parent | on HN ]

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by ajuc on Tuesday December 14, 2021 @11:06AM (#29553103)

Yup, the matrix for transforming hardware coordinates into software coordinates was saved in a config file using locale-specific number format. When user changed locale it remained in the previous format and if the change was English->German the matrix elements suddenly changed from 1.234 to 1234 :)

the details are here: https://github.com/jiixyj/libinput/commit/98793f6b435abf8c17...

The funny thing is we had the issue once - reported it and added a workaround (added special handling for these config files that used locale-specific calibration tables), then a year later we updated the linux version and had the issue the opposite way till we removed our workaround :) But it was even more fun cause it bricked the devices (kernel panic on libinput startup = restart loop forever) :)

[ Parent | on HN ]