Since the 70s or 80s, CPUs have gotten a lot faster while memory access has only gotten incrementally faster. This means that CPU manufacturers have put an increasingly sophisticated caching system on the CPU die to speed up access to main memory contents.
C's simple model allowing pointer arithmetic, aliased pointers that overlap to the same memory, etc, mean that compilers can't produce the optimal cache behavior on the CPU, because they can't reason definitively about when a given region of memory will be accessed based on the variables being accessed (pointer might have been modified or might overlap another one). They also have trouble invoking things like SIMD instructions which do math quickly on vectors. Fortran, with stricter rules about arrays, is more conducive to this.
Furthermore the vast majority of languages allow free aliasing, differing from C in what things pointers are allowed to point at and not in the number of pointers which may point at them.