The biggest downside I see is that it's non-portable, but the reality is that there's not all that many architectures out there to port to anyway (x86, ARM, MIPS probably covers 90%+) and for truly security-critical code having that level of control could be worth it. (This also avoids the "trusting the compiler" problem - an assembler is far easier to verify correctness of than even the simplest C compiler...)
... sort of. Chips themselves perform some optimizations.
http://blog.cryptographyengineering.com/2012/10/attack-of-we...
More recent stuff in this vein:
https://eprint.iacr.org/2014/435.pdf
(Yikes!)
There's a entire class of timing attacks that rely on the CPU cache - they would not exist if CPUs didn't do the optimization of storing local copies of limited sections of RAM.
Just like how a language that doesn't do array bounds checking and permits pointer math (read: a language where buffer overflows are a damn feature) isn't appropriate for security critical tasks.
Just like how a language that must be actively fought using clever hacks in order to prevent it from undermining your attempts to guard against common exploit vectors isn't appropriate for security critical tasks.