At some point however, the C# becomes so in-idiomatic or convoluted that it's just C in C# clothing. So one always needs to ask if it's time to just offload a task to a native library. It's convenient to have a single toolchain and not mess with a native toolchain though.
Especially since with native you're giving up AnyCPU, unless compiling the native parts for every platform you care about. I'd also argue that even unidiomatic C#'s semantics are less prone to UB than switching to C.
The optimal thing to do would be to memcpy several KB of data at a time from your heap-based memory, into your stack-allocated memory, do your processing in the optimized inner-loop, then commit the data back to the heap. I had to do this once to increase the throughout of a cluster of image processing servers.
I often scoff at the idea that C# isn't a good choice for performance-intensive tasks.