News for hackers. Stuff that matters.
Re:Debugging memory corruption: who the hell writes "2" into... by rectang on Tuesday December 31, 2024 @01:17PM (#42560596)
After trying and failing over several days to track down a squirrely segfault in a C project about 15 years ago, I taught myself Valgrind in order to debug the issue.

Valgrind flagged an "invalid write", which I eventually hunted down as a fencepost error in a dependency which overwrote their allocated stack array by one byte. I recall that it wrote "1" rather than "2", though, haha.

> Lesson learnt, folks: do not throw exceptions out of asynchronous procedures if you’re inside a system call!

The author's debugging skills are impressive and significantly better than mine, but I find this an unsatisfying takeaway. I yearn for a systemic approach to either prevent such issues altogether or to make them less difficult to troubleshoot. The general solution is to move away from C/C++ to memory safe languages whenever possible, but such choices are of course not always realistic.

With my project, I started running most of the test suite under Valgrind periodically. That took took half an hour to finish rather than a few seconds, but it caught many similar memory corruption issues over the next few years.

Re:Debugging memory corruption: who the hell writes "2" into... by jart on Tuesday December 31, 2024 @03:18PM (#42561500)
No the solution isn't to rewrite it in Rust. The solution is to have the option of compiling your C/C++ program with memory safety whenever things go loopy. ASAN, MSAN, and UBSAN are one great way to do that. Another up and coming solution that promises even more memory safety is Fil-C which is being made by Epic Games. https://github.com/pizlonator/llvm-project-deluge/blob/delug...
Re:Debugging memory corruption: who the hell writes "2" into... by kelnos on Tuesday December 31, 2024 @03:28PM (#42561591)
The solution is usually not to do a rewrite, but I think for greenfield projects we should stop using C or C++ unless there is a compelling reason to do so. Memory-safe systems languages are available today; IMO it's professionally irresponsible to not use them, without a good reason.

MSAN, ASAN, and UBSAN are great tools that have saved me a lot of time and headaches, but they don't catch everything that the compiler of a memory safe language can, at least not today.

Re:Debugging memory corruption: who the hell writes "2" into... by jart on Tuesday December 31, 2024 @03:55PM (#42561760)
Rust isn't standardized. Last time I checked, everyone who uses it depends on its nightly build. Their toolchain is enormous and isn't vendorable. The binaries it builds are quite large. Programs take a very long time to compile. You need to depend on web servers to do your development and use lots of third party libraries maintained by people you've never heard of, because Rust adopted NodeJS' quadratic dependency model. Choosing Rust will greatly limit your audience if you're doing an open source project, since your users need to install Rust to build your program, and there are many platforms Rust doesn't support.

Rust programs use unsafe a lot in practice. One of the greatest difficulties I've had in supporting Rust with Cosmopolitan Libc is that Rust libraries all try to be clever by using raw assembly system calls rather than using libc. So our Rust binaries will break mysteriously when I run them on other OSes. Everyone who does AI or scientific computing with Rust, if you profile their programs, I guarantee you 99% of the time it's going to be inside C/C++ code. If better C/C++ tools can give us memory safety, then how much difference does it really make if it's baked into the language syntax. Rust can't prove everything at compile time.

Some of the Rust programs I've used like Alacrity will runtime panic all the time. Because the language doesn't actually save you. What saves you is smart people spending 5+ years hammering out all the bugs. That's why old tools we depend on every day like GNU programs never crash and their memory bugs are rare enough to be newsworthy. Finally, the Rust community has a reputation for toxic behavior that raises questions about its the reliability of its governance.

Re:Debugging memory corruption: who the hell writes "2" into... by vvanders on Tuesday December 31, 2024 @09:52PM (#42563588)
You may want to refresh your familiarity with Rust, I haven't touched nightly in ages and much of what you mention doesn't really resonate with what I've seen in practice. Not saying the language doesn't have issues and things that aren't frustrating but in my experience unless you're going to go to the nines in testing/validation/etc (which is the first thing that's cut when schedules/etc are in peril) I've seen Rust code scale better than C++ ever did.

More tools in the C/C++ realm are always welcome but I've yet to see more than 50% of projects I've worked on be able to successfully use ASAN(assuming you've got the time to burn to configure them and all their dependencies properly). I've used ASAN, CBMC and other tools to good effect but find Rust more productive overall.

Re:Debugging memory corruption: who the hell writes "2" into... by saagarjha on Tuesday December 31, 2024 @11:04PM (#42563869)
> One of the greatest difficulties I've had in supporting Rust with Cosmopolitan Libc is that Rust libraries all try to be clever by using raw assembly system calls rather than using libc.

I’m sorry, this is coming from Justine “the magic syscall numbers are my god given right to use” Tunney?

Re:Debugging memory corruption: who the hell writes "2" into... by fc417fc802 on Tuesday December 31, 2024 @11:26PM (#42563949)
Seems like it depends entirely on context. I'd expect code which intends to be portable to use some sort of dynamically linked wrapper, even if that wrapper isn't libc.
Re:Debugging memory corruption: who the hell writes "2" into... by jart on Wednesday January 01, 2025 @07:20AM (#42565601)
It's the stated mission of my project. Rust's stated mission is memory safety.

So using raw system calls is a distraction for them.

Re:Debugging memory corruption: who the hell writes "2" into... by AlotOfReading on Tuesday December 31, 2024 @04:48PM (#42562110)
Ubsan is fantastic, but ASAN and the rest have serious caveats. They're not suitable for production use and they have a tendency to break in mysterious, intermittent ways. For example, Ubuntu 24.04 unknowingly broke Clang <=15ish when it increased mmap_rnd_bits. ASAN on Windows will actually check if you have ASLR enabled, disable it, and restart at entry. They interact in fun ways with LD_PRELOAD too.
Re:Debugging memory corruption: who the hell writes "2" into... by james_promoted on Tuesday December 31, 2024 @07:34PM (#42562980)
I'm on Clang 19 and still have a bunch of those sysctl commands sitting around.
Re:Debugging memory corruption: who the hell writes "2" into... by AlotOfReading on Tuesday December 31, 2024 @08:07PM (#42563134)
I'm not in a position to look up exactly when it was merged, but I'm pretty confident that shouldn't be needed anymore. The entry point on 19 should do the same restart juggling it does on Windows if the environment isn't correct for some other reason. I can double check later if you want to provide details.

I encountered the issue when our (not Ubuntu, not 24.04) LTS upstream backported security fixes that included the mmap changes without updating universe to include a clang version with the fixes. Any developers diligent enough to update and run sanitisers locally started seeing intermittent crashes.

Re:Debugging memory corruption: who the hell writes "2" into... by saagarjha on Tuesday December 31, 2024 @11:01PM (#42563867)
None of those sanitizers provide full memory safety for C/C++.
Re:Debugging memory corruption: who the hell writes "2" into... by ryao on Wednesday January 01, 2025 @04:22AM (#42564942)
That would not catch this. This is technically undefined behavior, but UBSAN does not have a check for this:

https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#...

What would catch this would be a C++ compiler warning. I filed bug reports against both GCC and LLVM asking for one:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263

https://github.com/llvm/llvm-project/issues/121427

Re:Debugging memory corruption: who the hell writes "2" into... by pjmlp on Tuesday December 31, 2024 @02:55PM (#42561318)
Similar experience, spending one week debugging memory corruption issues in production back in 2000, with the customer service pinging our team every couple of hours, due to it being on an high profile customer, has been my lesson.
Re:Debugging memory corruption: who the hell writes "2" into... by ch33zer on Tuesday December 31, 2024 @05:09PM (#42562229)
Doesn't C++ already support everything you need here? It supports the noexcept keyword which should have been used in the interface to this syscall. That would have prevented throwing callbacks from being used at compile time. My guess is that this is a much older syscall than noexcept though.
Re:Debugging memory corruption: who the hell writes "2" into... by cryptonector on Tuesday December 31, 2024 @06:05PM (#42562537)
I think the PAPCFUNC type needs to have noexcept. Wrapping a function typedef in extern "C" does not make it imply noexcept IIRC.

It would also help if the APC docs documented that APCs must not throw.

Re:Debugging memory corruption: who the hell writes "2" into... by ch33zer on Tuesday December 31, 2024 @07:42PM (#42563017)
This would actually be a nice change (but probably very breaking) for c interfaces called from a c++ context to be implicitly noexcept.
Re:Debugging memory corruption: who the hell writes "2" into... by ryao on Wednesday January 01, 2025 @04:18AM (#42564930)
It is a C interface. It is implicitly noexcept. I filed bugs against both GCC and LLVM requesting warnings when someone passes a non-noexcept C++ function pointer to either a C function or to a C++ noexcept function:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263

https://github.com/llvm/llvm-project/issues/121427

Perhaps the C++ standards committee should specify that doing this should cause a compiler failure, rather than a warning.

Re:Debugging memory corruption: who the hell writes "2" into... by cryptonector on Wednesday January 01, 2025 @11:56AM (#42567256)
Thanks for checking, clarifying, and filing those bugs!
Re:Debugging memory corruption: who the hell writes "2" into... by masspro on Tuesday December 31, 2024 @05:34PM (#42562358)
noexcept doesn’t prevent any throws at compile-time, it basically just wraps the function in a `catch(...)` block that will call std::terminate, like a failed assert. IMHO it is a stupid feature for this very confusion.
Re:Debugging memory corruption: who the hell writes "2" into... by ch33zer on Tuesday December 31, 2024 @07:38PM (#42562995)
This was true until c++17. It was changed in 17 to make noexcept part of the function type meaning a noexcept(false) function can't be used in a context where a noexcept is needed as they're unrelated types. I don't know if compilers actually implement this but according to the standard it should be usable.

https://en.cppreference.com/w/cpp/language/noexcept_spec

Re:Debugging memory corruption: who the hell writes "2" into... by masspro on Tuesday December 31, 2024 @07:56PM (#42563080)
Yes this helps specifically when passing functions as pointers or something like std::function (edit: or overriding methods), it will at least inform the developer that they need to add noexcept to the function declaration if they want to use it there, and hopefully due to that they recursively audit the function body and anything it calls for exceptions. And hopefully all future developers also notice the noexcept and keep up the practice. But it changes nothing about checking plain function calls. So I think adding this to the function type helps some cases but still does not move noexcept toward the behavior most people want/expect.

This just feels important to point out because this feature is 15 years old and still commonly misunderstood, and each time people are wanting the same thing (actual compile-time prevention of `throw`) which it is not.

Re:Debugging memory corruption: who the hell writes "2" into... by ryao on Wednesday January 01, 2025 @04:16AM (#42564918)
If the compiler would just issue warnings in the following cases, then noexcept would be useful for preventing this problem:

https://godbolt.org/z/rjPfYjnzf

https://godbolt.org/z/14ocshsE5

I filed bugs against both GCC and LLVM requesting warnings:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263

https://github.com/llvm/llvm-project/issues/121427

Re:Debugging memory corruption: who the hell writes "2" into... by saagarjha on Tuesday December 31, 2024 @11:01PM (#42563863)
Valgrind is neat but it wouldn’t help here, unfortunately
Re:Debugging memory corruption: who the hell writes "2" into... by ryao on Wednesday January 01, 2025 @04:53AM (#42565043)
Switching to memory safe languages would not necessarily prevent this issue because this is an undefined behavior issue, not a memory safety issue.

Pointers to C++ functions that can throw exceptions should not be passed to C functions as callback pointers. Executing the exception in the callback context is undefined behavior, since C does not support stack unwinding.

Presumably, any language that has exception handling would have an issue on Windows when doing select() in one thread and QueueUserAPC() in another thread to interrupt it with a callback function that throws an exception. What happens then depends on how the stack unwinding works.

No programming language can avoid this because they need to use the underlying operating system's system calls in order to function. They also need to use the C functions in ntdll to do system calls since Microsoft does not support doing system calls outside of calling the corresponding functions in ntdll and friends.