News for hackers. Stuff that matters.
Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by dig1 on Tuesday December 14, 2021 @06:59AM (#29550715)
Not only Facebook, but the whole industry, if I may say. For example, I had a client who could not get rid of weird JVM GC pauses that would come from nowhere, after which it would render the application unusable for some time. The cheapest solution (as they could not scale it to more machines because it would just replicate the matter) was to add a custom code that would ping systemd watchdog every X seconds. If that stopped, systemd would restart the app.

I think there are two problems with "hard core" debugging: 1) these things are costly and will usually pull in the best devs (so no one will be working on that "important" UI-related ticket that should have been finished a week ago /s). 2) for the management team, spending 10-15 hours fixing one-line or/xor/bit-flip magic stuff doesn't make much sense, and I'd usually get responses like: "oh, it was THAT simple?". Yeah, right, none of you'd seen I've read that Intel manual twice and re-evaluated my math knowledge multiple times over the last two nights :D

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by oblio on Tuesday December 14, 2021 @07:20AM (#29550840)
IIS literally has functionality built-in to restart its subprocesses or whatever they're called, if the memory usage goes of X or if the subprocess crashes.

And yes, it is used in production in very big companies.

The main thing that was monitored was actually the restart rate/memory increase rate, i.e. a restart every 6 hours: ok, a restart every 30 minutes, not ok.

Re:Drgn: How the Linux kernel team at Facebook debugs the ke... by nvr219 on Tuesday December 14, 2021 @10:11AM (#29552368)
If it's stupid and it works it ain't stupid.