As I understand it, the windows minidump contains the stack data for each thread. This is more than the stack trace: It also contains local variables. Today, most interesting things are probably pointers outside the minidump, but it is not uncommon in e.g. C to create a local array and dump some strings in them. Then the problem comes back.
Yes, when I wrote "which is basically the stack for each thread" I did mean the raw bytes of the stack. I see now that wasn't clear. I was writing on mobile and apologize for the confusion.
Unwinding then occurs later by walking through the stack memory, which is difficult to impossible unless either: a) the stack contains frame pointers; or b) you have unwind data. Without either of those the unwinder uses heuristics to guess at return addresses on the stack, but that's very unreliable and often leads the unwinder into a loop till it gives up.
No apologies needed. I am thankfull for your detailed, interesting post.
Re:Debugging a Futex Crash
by TazeTSchnitzel on Wednesday August 16, 2023 @07:23PM
(#37155197)
Thanks for this detailed overview of the crash dump landscape!
On Linux and Android, you can similarly do unwinding using something like libunwind on the device. However, the traditional solution is that a "core" file is dumped and then is unwound later.
Years ago Microsoft invented a format called the "minidump" which is basically the stack for each thread, CPU registers, and other data. This format was adopted by Google Breakpad, and which ended up being used by both Firefox and Chrome for their crash reporting. The minidump format is often used when for whatever reason, unwinding can't be done on the device. It's a fairly flexible binary format.
Breakpad can perform sanitation before sending the minidump off the device, but this can be hit or miss.
Sentry (and probably other services) support minidumps among other formats, but minidumps are a PITA and I'll bet they hate supporting them. It's a large file despite its name, and it's expensive, complicated, and risky to unwind on the backend. The only open-source minidump unwinder used to be fairly esoteric C++ code, but there's now a rust-based unwinder too.
Breakpad actually came up with something called a microdump, which is an even smaller version of a minidump that's text based instead of binary.
Both Apple and Google make mobile crash reporting way harder than it need be. Both Android and iOS have their own crash handlers that can perform on-device out-of-process unwinding, but the reports are not available to the app itself.
You only get whatever reports Apple and Google feel like showing you after they get uploaded to their systems. So apps have to embed their own crash handlers. Because those handlers are operating in the same memory space as the app which just crashes, their operation is fraught with peril. At least on iOS, unwinding is trivial due to mandatory frame pointes, and symbolication is easy because Apple makes iOS symbols available.
Android is a cluster fuck. You may or may not have frame pointers. So unwinding on the device is hit or miss. You also have to deal with both native code and ART (Android Run Time) code. Tracing back through the runtime is non-trivial to impossible. And you will almost definitely not have debugging symbols. So good luck with symbolication. The lack of debugging symbols also make off-device unwinding basically useless, so a minidump isn't that helpful.
I designed and built the mobile crash reporting solution for my company about a decade ago, though we're currently migrating to a SAAS solution.