Everything in C operates according to the "as if" rule. In other words, the resulting program merely needs to behave as if the result was executed according to the spec. How execution happens is left entirely up to the compiler.

C also doesn't define "the stack" in any way. And of course alloca() isn't part of C at all, but on the systems where it exists, it's not very specific about just where it allocates stuff, just that it's on that nebulous "the stack".

Given the above, I believe you cannot write a conforming program that can detect the difference between alloca(0) returning a region of 0 bytes at the end of the stack and alloca(0) returning something else. Since no conforming program can detect it, the "as if" rule allows the return value to be considered to be anything at all.

The program in question is not a conforming program in the first place. It's the Ruby interpreter and it does all kinds of low-level stuff. alloca(0) is called from the garbage collector in order to detect the end of the stack so that the garbage collector can scan the stack for pointers. The code assumes that it's running on a system where there is a stack at all, which is pretty much all systems nowadays.

Of course it's not a conforming program. That's rather the point: as a non-conforming program, the compiler is allowed to apply optimizations which may behave differently from what the programmer wants it to do. That this code works on one compiler and fails on another doesn't make it a compiler bug, though. It merely means that this code relies on the compiler behaving in a certain way which isn't actually mandated.