By this reasoning, though, any behavior is defined as long as we box it off into a library.
(By "defined" here, I mean "ISO C defined" not "declared to be working, and maintained in that order by a library vendor").
> If Boehm GC were written in assembly
Every instance of undefined behavior that doesn't prevent translation results in machine code. We can infer its behavior from the instruction manual, and write assembly source which produces the same code. Therefore, there is no such thing as undefined behavior.
A C program doesn't stop being a C program because it calls into a BLAS library that happens to be implemented in Fortran or some low-level library implemented in assembly.
Again: what property of the interface of Boehm GC breaks any revision of the ISO C standard?
A C program also stops being an ISO C program when it calls a function that is not in the standard C library and that is written in Fortran. However, that Fortran code can be well-defined Fortran, and the call can be taking place through a C-Fortran calling mechanism documented by the compiler suite. So that even though we don't have a definition of behavior entirely coming from ISO C (so that the program is undefined as far as ISO C is concerned), we can piece together a definition of behavior from ISO C and additional sources, like the Fortran standard and the reference manual for the toolchain.
We don't have any such definition of behavior with Boehm GC. The behavior is empirically validated. No compiler document says that it's okay to scan memory looking for pointers, as a documented extension.
The interface to Boehm GC is that you call GC_INIT(), then you get some pointers from GC_MALLOC() and GC_REALLOC(). Nothing in ISO C requires that a function returning a pointer must guarantee that pointer remains valid forever. Which section number of any of the ISO C standards do you think Boehm GC violates?
The implementation details of the Boehm GC certainly exploit implementation-defined behavior and there's likely some undefined behavior there, too. However, those are implementation details of the library itself. If Boehm GC were written in assembly, then it could function exactly the same and ISO C would be irrelevant to its implementation.