Fast Slow
Runtime (RDTSC) [s] | 4.912542e-01 | 6.476124e-01
RESOURCE_STALLS_ANY | 135,743,115 |1,298,835,857
IDQ_DSB_UOPS |1,441,709,567 | 97,635
LSD_UOPS | 197,902,567 |1,999,996,841
IDQ_MITE_UOPS | 360,547,507 | 82,354
INSTR_RETIRED_ANY | 1600095725 | 1600095769
CPU_CLK_UNHALTED_CORE | 1662620966 | 2192914751
First, RESOURCE_STALLS_ANY is picking up some important difference. Slow has lots, Fast has fewer. Breaking these down to see what subtype it is might give an exact answer: https://download.01.org/perfmon/index/skylake.html. It's not RESOURCE_STALLS_SB that changes, but the others don't seem to be predefined.
LSD_UOPS is the number of µops delivered by the "Loop Stream Detector", which is the smallest and fastest cache for instructions. It's very small, and only works for tiny loops. Usually this is where you want the µops to stream from, but in this case using it results in a slower final result.
IDQ_DSB_UOPS is the number of µops delivered from the "Decode Stream Buffer". This is usually slightly slower than the "Loop Stream Buffer", but in this case using it produces a faster result. This is large enough to hold a reasonably sized function.
IDQ_MITE_UOPS is the "legacy" instruction decoder. This is the normal path by which instructions are decoded into µops. It's always used the first time instructions are encountered, but for hot loops and hot functions the results can be cached. Usually you'd like to see less of this, but here using the legacy decoding path gives a faster result.
What does this mean? It probably means that something about the alignment combined with the jump defeats what would normally be the fastest µop cache. This isn't that strange in itself, but the oddity is that for some reason, this causes an overall faster result.
Why? I'd thought it was something to do with "store forwarding", and it still might be, but I haven't found any counters that directly point to that as being the problem. Instead, I think it might have to do with the processor getting too far ahead on the reads and stores, and then having to redo them once it realizes that the data is out of date. Somehow having fewer instructions available in the reorder buffer (because they are coming in from a slightly slower decoder) improves overall performance. But I don't yet understand the specifics.
I'm going to stop for the night, but I'd love to wake up to someone revealing the "true" answer!
In any case, I have to admit being wrong; port 6 has nothing to do with it, though the general principle of stalling the frontend to prevent the backend being overloaded still holds. For example, here's how to get the speedup without any change in control flow:
mov rcx, 400000000-1
align 32
nop
up:
align 32
add rcx, [counter]
align 32
dec rcx
align 32
jnz up
ret
As to why this actually makes things faster, I don't know. There are many more stalls owing to the reservation station being full in the tight loop:
1,348,491,506 resource_stalls_any
1,694,613,980 uops_issued_stall_cycles
759,763,767 uops_executed_stall_cycles
994,974,421 uops_retired_stall_cycles
109,300,935 resource_stalls_any
120,303,318 uops_issued_stall_cycles
164,322,430 uops_executed_stall_cycles
15,834,456 uops_retired_stall_cycles
756,686,062 uops_dispatched_port_port_0
782,579,771 uops_dispatched_port_port_1
279,387,822 uops_dispatched_port_port_2
314,214,218 uops_dispatched_port_port_3
1,994,313,938 uops_dispatched_port_port_4
807,514,909 uops_dispatched_port_port_5
1,050,884,205 uops_dispatched_port_port_6
209,316,457 uops_dispatched_port_port_7
640,957,968 uops_dispatched_port_port_0
637,621,266 uops_dispatched_port_port_1
399,224,647 uops_dispatched_port_port_2
400,205,411 uops_dispatched_port_port_3
1,610,294,604 uops_dispatched_port_port_4
643,780,748 uops_dispatched_port_port_5
848,142,785 uops_dispatched_port_port_6
470,700 uops_dispatched_port_port_7
Section 2.3.2.4 describes it for Sandy Bridge (and the rules don't seem to have changed in later generations):
The loops with the following attributes qualify for
LSD/micro-op queue replay:
• Up to eight chunk fetches of 32-instruction-bytes.
• Up to 28 micro-ops (~28 instructions).
• All micro-ops are also resident in the Decoded ICache.
• Can contain no more than eight taken branches and
none of them can be a CALL or RET.
• Cannot have mismatched stack operations. For example,
more PUSH than POP instructions.I haven't been following that bug closely, but I think I'm OK on this machine since hyperthreading is turned off? This a a benchmarking machine, so I have to be careful about making changes that will affect historical results.
the general principle of stalling the frontend to prevent the backend being overloaded still holds
I think I agree that this is what is happening. Why it's necessary, and the exact mechanism by which it helps are places that I'm uncertain about. I presume you are seeing the instructions showing up as IDQ_MITE_UOPS?
From https://www.intel.com/content/www/us/en/architecture-and-tec... Section 2.3.2.2, here's some more limitations of the Decoded ICache:
The Decoded ICache consists of 32 sets. Each set contains
eight Ways. Each Way can hold up to six micro-ops. The
Decoded ICache can ideally hold up to 1536 micro-ops.
The following are some of the rules how the Decoded ICache
is filled with micro-ops:
• All micro-ops in a Way represent instructions which are
statically contiguous in the code and have their EIPs
within the same aligned 32-byte region.
• Up to three Ways may be dedicated to the same 32-byte
aligned chunk, allowing a total of 18 micro-ops to be
cached per 32-byte region of the original IA program.
• A multi micro-op instruction cannot be split across Ways.
• Up to two branches are allowed per Way.
• An instruction which turns on the MSROM consumes
an entire Way.
• A non-conditional branch is the last micro-op in a Way.
• Micro-fused micro-ops (load+op and stores) are kept as
one micro-op.
• A pair of macro-fused instructions is kept as one micro-op.
• Instructions with 64-bit immediate require two
slots to hold the immediate.
Focus on port 4, the only port available for memory stores.
Yes, we'd expect 4B, and we instead see 1.6B for the fast case and 2.0B for the slow. My version above is at 1.4B, and I'd guess is proportionally faster. Have you seen tricks that reduce this to some lower number?
We'd also expect P2 plus P3 to equal 4B? It's interesting that these are closer, but still quite a bit over.
By dispatching fewer instructions that we can handle per cycle, there is less speculation and thus less wasted work?
While there is less waste in terms of power efficiency, are we actually seeing P4 be a bottleneck? It's hard to tell as the total cycles also goes up, but I think it's still at 90% rather than 100% utilization.
So while it looks bad, I'm not sure that anything will necessarily go faster if we reduce the number further. It may be that the maximum difference is whether we get a 4 cycle turnaround on the store-load forwarding versus a 5 cycle.
No, the legacy decoder is virtually unused. This loop was designed to take advantage of the rules of the DSB. If you have the LSD enabled I don't quite know what the effect is.
EDIT: Sorry, just realized I screwed up translating AT&T to Intel---add rcx, [counter] should clearly be add [counter], rcx.
> It may be that the maximum difference is whether we get a 4 cycle turnaround on the store-load forwarding versus a 5 cycle.
This is my guess as well, though I have nothing to show for it. It's actually possible to bring it down to 400000000 uops by delaying the loop even further with the same technique, but it won't speed anything up. The 4-cycles-per-iteration barrier seems insurmountable, which makes sense.
OK, think I see now. The DSB path is only able to inject one "way" per cycle, and through NOP padding you are able to ensure that each "way" only contains a single instruction? While I can find clear descriptions of what can fit in a "way", I haven't found a clear statement of the "one way per cycle" restriction, although it would explain observed behavior.
Practically, it seems like the effect would be the same if one "defeated" the DSB and came up with a similar strategy for causing the legacy MITE to trickle in instructions. The overall goal is to restrict speculative execution of loads and stores in cases where one knows that the speculation is going to fail.
You'd think that there would be an MFENCE/LFENCE/SFENCE way of solving this more directly, but I haven't found it yet. Or maybe just a LOCK? Maybe a creative false dependency to keep the processor from getting to far ahead? Interesting. Thanks for helping explore this.
mov rcx, 400000000-1
up:
nop
nop
nop
add [counter], rcx
nop
nop
nop
dec rcx
nop
nop
nop
jnz up
ret
I'm working with a slightly modified loop that counts down rather than up, so that the loop has slightly fewer instructions. I've switched to using an "add to memory" to reduce the instruction count further. None of these seem to directly affect the speed of the loop, but here's the fastest loop I have:
On Skylake, it executes in 0.497 s. If I remove the "jump to the next instruction", it slows to 0.646 s. This agrees with your explanation. But here's the part I don't understand, and that may undercut your theory: if I change the first line to be ".p2align 4" (that is, if I increase the minimum alignment from 8 to 16) the speed is the same whether or not I have the extra jump. Even more confusingly, it's always the slower speed!Here's what port usage looks like for the two different cases (both with the same instructions, just different alignments):
From the first line we see that both Fast and Slow have the same number of instructions executed (which makes sense, since both are executing the same instructions), but down below we see very different numbers of µops dispatched! I'm not sure what to make of this, but I think this shows that it is not (solely?) a front end issue.