Using things like stackalloc, malloc, and local-only references, It is possible to implement algorithms in C# which can run at blazing-fast speeds, sometimes hundreds of times faster than idiomatic C#. C#-style arrays would not make the cut.
The optimal thing to do would be to memcpy several KB of data at a time from your heap-based memory, into your stack-allocated memory, do your processing in the optimized inner-loop, then commit the data back to the heap. I had to do this once to increase the throughout of a cluster of image processing servers.
I often scoff at the idea that C# isn't a good choice for performance-intensive tasks.
At some point however, the C# becomes so in-idiomatic or convoluted that it's just C in C# clothing. So one always needs to ask if it's time to just offload a task to a native library. It's convenient to have a single toolchain and not mess with a native toolchain though.
Especially since with native you're giving up AnyCPU, unless compiling the native parts for every platform you care about. I'd also argue that even unidiomatic C#'s semantics are less prone to UB than switching to C.
It does (The JIT that is), but it also often does not. For example a regular for loop does not bounds check anything. The newer x64 JIT ("RyuJIT") is a whole lot smarter than the legacy x86 jit was.
I imagine that once you start producing proper native binaries using whatever backend you want instead of a JIT that has a requirement of being FAST in addition to making code that is FAST, you can get even more optimization done.
Also, bounds check are becoming cheaper and cheaper as memory fetches become (relatively) more expensive with every hardware generation.
C++ code is usually faster than C# code but bounds checking is probably not the biggest culprit. If you use "idiomatic C#" with reference types, most operations have a lot of memory fetch overhead with pointer chasing etc.
It's possible to write pretty efficient C# code, and it can still look pretty nice. You end up using arrays more, structs more, obviously worrying about AoS vs SoA more. You should also look at the new ImmutableArray types that look like List<T> but have effectively no overhead compared to a plain array. After that comes Span<T>, which makes even more exotic things possible.