Interesting. Thanks for the links. Isn't C's stack pretty much tied to the hardware? Curious what you mean by "If C's stack is fixed"? How could that be implemented, changing the run time?
Re:Teaching C
by nickpsecurity on Wednesday May 11, 2016 @10:36PM
(#11680969)
You change the compiler to emit different things like a reverse stack or whatever your protection model is. Far as implementation, they describe it in p5 (PDF p7) of paper above (not MULTICS paper). It's actually brilliant now that I read it as the naive thing they avoid is, IIRC, what the other academics did on Linux/GCC. The performance overhead hit 10% easily due to x86's stack approach. I think worst-case was even higher. This team effectively tricks the CPU with simple instructions (eg addition/subtraction) without invoking memory to get it to worst-case of 2%. Clever.
Note: I'm not saying this is sufficient to stop stack smashing. Just that reverse stacks are a better idea than the ludicrous concept of making unknown amount and quality of data flow toward the stack pointer. Definitely reduced risk a bit but how much takes more assessment.
https://www.acsac.org/2002/papers/classic-multics.pdf
Really old stuff. Relevant quote: "Third, stacks on the Multics processors grew in the positive direction... if you actually accomplished a buffer overflow, you would be overwriting unused stack frames rather than your own return pointer, making exploitation much more difficult."
I can't find the original paper showing the penalty on x86/Linux. However, this one does the same thing for different reasons with many details:
http://babaks.com/files/TechReport07-07.pdf
Key point: "The direction of stack growth is not flexible in hardware and almost all processors only support one direction. For example, in Intel x86 processors, the stack always grows downward and all the stack manipulation instructions such as push and pop are designed for this natural growth."
So, on such an architecture, you can't directly use the stack operations to do the job: must implement extra instructions without hardware acceleration. The stack on x86 is effed-up and insecure by design. If C's stack is fixed, there's still a mismatch between it and x86 ASM. Itanium at least provided stack protection among other security benefits.