News for hackers. Stuff that matters.
Re:Debugging a Futex Crash by no_time on Wednesday August 16, 2023 @03:04AM (#37143685)
Do these crash reporting tools have mechanisms in place to not leak sensitive data in the coredumps? Would be awkward if a firefox coredump upload would leak my session cookies.
Re:Debugging a Futex Crash by hyperman1 on Wednesday August 16, 2023 @03:39AM (#37143897)
That's almost impossible, and one of the reasons automatic crash reporting is problematic.

It is an issue with e.g medical data: Doctors using mswindows or any browser based tool can at any moment export random medical data to god knows which country. In the EU, it is probably illegal to even look at it if you receive it, and you don't know which dreports even have this issue.

Afaik, the only reason things aren't crashing down is total non-enforcement by all authorities. Which raises interesting question about their will and capability to enforce e.g. the medical secret at all.

Re:Debugging a Futex Crash by TazeTSchnitzel on Wednesday August 16, 2023 @04:34AM (#37144281)
Automatic crash reporting doesn't necessarily contain sensitive info, it depends on how much detail is provided. When I get a crash on macOS, the reporter lets me the report before it's sent, and as far as I can tell it's just plaintext stacktraces of active threads, plus a handful of other details.
Re:Debugging a Futex Crash by rocqua on Wednesday August 16, 2023 @08:04AM (#37145761)
As I understand a 'core-dump' is not just a stack trace, but also contains some of the arguments passed in the functions in the stack-trace. Those can easily contain sensitive information. Especially if the heap is also included.
Re:Debugging a Futex Crash by TazeTSchnitzel on Wednesday August 16, 2023 @08:09AM (#37145805)
Yes, a core dump is sensitive. It concerns me if those are ever sent over the wire automatically.
Re:Debugging a Futex Crash by js2 on Wednesday August 16, 2023 @10:59AM (#37148117)
macOS (and iOS) do the stack unwinding on the device. The "backend" then only has to perform symbolication (mapping addresses to symbol names). This is how both the built-in macOS/iOS crash reporter work, as well as the third-party PLCrashReporter and KSCrash crash handlers.

On Linux and Android, you can similarly do unwinding using something like libunwind on the device. However, the traditional solution is that a "core" file is dumped and then is unwound later.

Years ago Microsoft invented a format called the "minidump" which is basically the stack for each thread, CPU registers, and other data. This format was adopted by Google Breakpad, and which ended up being used by both Firefox and Chrome for their crash reporting. The minidump format is often used when for whatever reason, unwinding can't be done on the device. It's a fairly flexible binary format.

Breakpad can perform sanitation before sending the minidump off the device, but this can be hit or miss.

Sentry (and probably other services) support minidumps among other formats, but minidumps are a PITA and I'll bet they hate supporting them. It's a large file despite its name, and it's expensive, complicated, and risky to unwind on the backend. The only open-source minidump unwinder used to be fairly esoteric C++ code, but there's now a rust-based unwinder too.

Breakpad actually came up with something called a microdump, which is an even smaller version of a minidump that's text based instead of binary.

Both Apple and Google make mobile crash reporting way harder than it need be. Both Android and iOS have their own crash handlers that can perform on-device out-of-process unwinding, but the reports are not available to the app itself.

You only get whatever reports Apple and Google feel like showing you after they get uploaded to their systems. So apps have to embed their own crash handlers. Because those handlers are operating in the same memory space as the app which just crashes, their operation is fraught with peril. At least on iOS, unwinding is trivial due to mandatory frame pointes, and symbolication is easy because Apple makes iOS symbols available.

Android is a cluster fuck. You may or may not have frame pointers. So unwinding on the device is hit or miss. You also have to deal with both native code and ART (Android Run Time) code. Tracing back through the runtime is non-trivial to impossible. And you will almost definitely not have debugging symbols. So good luck with symbolication. The lack of debugging symbols also make off-device unwinding basically useless, so a minidump isn't that helpful.

I designed and built the mobile crash reporting solution for my company about a decade ago, though we're currently migrating to a SAAS solution.

Re:Debugging a Futex Crash by hyperman1 on Wednesday August 16, 2023 @03:46PM (#37152645)
As I understand it, the windows minidump contains the stack data for each thread. This is more than the stack trace: It also contains local variables. Today, most interesting things are probably pointers outside the minidump, but it is not uncommon in e.g. C to create a local array and dump some strings in them. Then the problem comes back.
Re:Debugging a Futex Crash by js2 on Wednesday August 16, 2023 @05:22PM (#37154033)
Yes, when I wrote "which is basically the stack for each thread" I did mean the raw bytes of the stack. I see now that wasn't clear. I was writing on mobile and apologize for the confusion.

Unwinding then occurs later by walking through the stack memory, which is difficult to impossible unless either: a) the stack contains frame pointers; or b) you have unwind data. Without either of those the unwinder uses heuristics to guess at return addresses on the stack, but that's very unreliable and often leads the unwinder into a loop till it gives up.

Re:Debugging a Futex Crash by hyperman1 on Friday August 18, 2023 @03:55PM (#37181219)
No apologies needed. I am thankfull for your detailed, interesting post.
Re:Debugging a Futex Crash by TazeTSchnitzel on Wednesday August 16, 2023 @07:23PM (#37155197)
Thanks for this detailed overview of the crash dump landscape!
Re:Debugging a Futex Crash by ishitatsuyuki on Wednesday August 16, 2023 @03:28AM (#37143841)
Unfortunately, removal of PIIs also means less data/cues for hard-to-diagnose bugs (it can even range to CPU bugs like [1]). For Firefox, potentially PII-containing memory dumps are still sent to the server, but they're only viewable by authorized personnel [2].

There are also rule-based PII removal methods as used by Sentry minidump processing and described at [3].

[1]: https://news.ycombinator.com/item?id=37063459

[2]: https://crash-stats.mozilla.org/documentation/protected_data...

[3]: https://docs.sentry.io/product/data-management-settings/scru...

Re:Debugging a Futex Crash by planede on Wednesday August 16, 2023 @04:19AM (#37144157)
Found this for coredumps and minidumps:

https://support.backtrace.io/hc/en-us/articles/360040517131-...

So there are at least some efforts to do it. IMO it can only ever be best effort, but it doesn't mean that it's not worth doing.