Any reasonable compiler will implement x%256 using bitwise arithmetic, but C semantics require that x == d*(x/d) + (x%d), which means that if x is negative (x%256) must be negative, while (x&0xff) will be positive.
This difference means that the AND is still faster than the MOD when used on signed integers, if the compiler is not smart enough to prove that the input will never be negative.
For example, with gcc 4.7.3 the resulting assembly is
movl %edi, %edx
sarl $31, %edx
shrl $24, %edx
leal (%rdi, %rdx), %eax
andl $255, %eax
subl %edx, %eax
It's essentially the same[0] as `prndindex = (prndinex+1) % 256`, except bitwise AND takes less effort to compute.
Modulo is essentially[1]: `mod = a - b * (a / b)` unless the compiler can manage to use bitwise shifting/masks instead.
Back when Doom was popular, the bitmask would have been much more efficient.
[0] http://stackoverflow.com/q/3072665/2967113 [1] I'm not 100% positive that's how it's implemented with an IDIV instruction.