In this case though, C doesn't have garbage collection or OO features, and (I believe) has unspecified behavior for arithmetic exceptions. I assume Fortran does the same (since it's comparably fast), so I guess I still don't really see where people expect the difference to come from.
As for what the differences are, try talking to a compiler engineer. The better ones love how Fortran features like the argument aliasing rule for 100% of code makes generating fast code easier. In fact that's one of my tests for compiler engineers in interviews; if they've worked on a C/C++/Fortran compiler and can't say anything good about the challenges and opportunities of Fortran optimization relative to C/C++, then they haven't really drilled down on performance issues!
Some language features close the door for optimizations. E.g. the ability to dynamically add or remove methods to an object/class can make method dispatch more or less expensive.
Finally, some languages have undefined behavior (e.g. what happens when there's an overflow or a division by zero) while other languages want specific behavior for the edge cases. This all affects the performance of the resulting code, even for basic arithmetic operations.