Not necessarily. In the early days of the P6 core, I found a simple loop which would take either 4.0 or 5.33 cycles depending on the internal CPU state -- exactly the same code, but running with an iteration count of 10^6 would take either 4000000 cycles or 5333333 cycles.
I had the good fortune to talk to a member of the team which designed the P6 core, and we concluded that this was due to a quirk of the out-of-order scheduler: In order to reduce circuit depths, the mechanism for finding and dispatching runnable uops wouldn't always dispatch the earliest available uops, and so -- depending on how the incoming uops were aligned within an internal buffer -- the P6 core would sometimes take a stream of instructions which could be executed in-order without pipeline stalls and run them out-of-order with less ILP.
In short, OOO cores are weird and horribly complicated and completely untrustworthy where performance is concerned.
Yep. There's a great talk about this by Cliff Click, called A Crash Course in Modern Hardware[1] that I would recommend to everyone. I am no hardware expert, so that talk really enlightened me.
Regarding the issue at hand, I remember Doug Lea saying[2] that some new Intel processors may recognize a loop as the OSs idle loop and power down the core. That's why he computes random numbers in busy-wait loops.
[1] http://www.infoq.com/presentations/click-crash-course-modern...
[2] http://emergingtech.chariotsolutions.com/2013/04/phillyete-s...
In order comes back during strange times. Sun switched back to in-order for Niagara, for instance.
Bottom line is you just have to think about GPUs in a slightly different way - they do have disadvantages (limited register/cache/memory per thread since you need so many of them), but things like branching aren't that big of a deal.
One interesting experiment to try would be to realign the loop to 0..31 mod 32, and see how/if the behavior changes.
Both loops should fit, but I think the same limits for micro-op density apply. The Optimization Manual says the Loop Stream Detector requires that "All micro-ops are also resident in the Decoded ICache". I took this to mean if something can't be cached by the Decoded ICache, it can't be streamed by the LSD.
Core2 had an 18 x86 instruction "loop stream detector".
Nehalem went to a 28 u-op (not instruction) LSD.
Sandy Bridge has a 1500 u-op "decoded instruction cache" plus a 28 u-op LSD.
Haswell is the same as SB, but doubles the u-op LSD to 56 if hyperthreading is disabled.
The confusing part is why there is still a tiny loop stream detector in addition to the much larger decoded iCache. Perhaps there is a small additional power savings for code that fits the smaller LSD?
But in general, you no longer have to worry about instruction decoding speed if your loops are less than 1500 u-ops. With some caveats: https://www-ssl.intel.com/content/dam/www/public/us/en/docum...
For example, in Intel processors with a u-op cache, a 32-byte fetch group that generates more than 18 u-ops has to go through the x86 decoder and cannot stream from cache.[1] The call instruction, being a pretty big instruction with that 32-bit immediate displacement, might force the loop onto two 32-byte lines, reducing the u-op count in each line below 18 and allowing the loop to stream from the u-op cache instead of going through the decoders.
In general, if adding instructions makes a loop go faster, it's some sort of alignment (maybe code layout is a better term?) issue. Intel CPU's have limits on things like the number of branch predictions per fetch group, number of decoded instructions per fetch group, etc. Some x86 CPUs can only track 3 branches in a 16-byte group, while it might be possible to encode 4 in unusually dense code. In such cases, the extra branch just won't be tracked by the predictor. Seemingly superfluous code can force some of the branches into a different 16-byte group, improving overall performance. (This can't be the case here, because there's just one branch, but it can pop up in branch-heavy code).
EDIT: Someone makes in the comments makes a good point that in any case, both loops should fit in the loop buffer, which is after the u-op cache, and so wouldn't suffer from the 18-uop limit mentioned above.
[1] See: http://www.realworldtech.com/sandy-bridge/4