Core dumps are great, especially for the occasional odd bug that occurs only in long running processes. However, the ridiculous systemd default of attempting and failing to capture the kernel core dump into the binary journal completely ruins this. Only small, uninteresting core dumps are ever captured; large ones are silently discarded because the journal is not allowed to grow that big.
Yeah, when I'm crashing and want to keep cores, I'll temporarily modify /proc/sys/kernel/core_pattern to prevent systemd from eating them. I sometimes want them suppressed, sometimes want them written out, and never want them to go to systemd.
One issue is that that sysctl is not a namespaced one.
> Is this because inspecting core files also sucks, since it's a snapshot of a point in time
Yes. Something is wrong, but you can't tell how you got into that state.
Right.
If a core file is all you've got, then you do your best (and there are some nifty researchy tools that help with that). But if you could have a full recording instead of the core file, then of course you'd want the recording, because that dramatically increases the probability you'll be able to debug the bug (to near 100%, if the bug is in the code you've recorded).
"The only valid use of a debugger is to analyze deadlocks in post Mortem core dumps. Interactive debugging is the greatest time waste. Write automated tests for your debug hypotheses instead." (Peter Sommerlad)
In my embedded work I avoided using debuggers because setting up remote debugging can be a PITA. I figured out a long time ago that building remote debug setup into my workflow will save oodles of time in the end. Still, I find myself using plain old logging more often than not - the bug may not be in C or C++, it may be on the JavaScript run in the interpreter embedded in that C++ application.