Re:Intel i7 loop performance anomaly
by edwintorok on Tuesday December 03, 2013 @04:34PM
(#6843357)
[can't edit previous post]
In fact replacing the nopw that is used for alignment by a prefetch instruction gives me something slightly even faster:
00000000004004e0 <prefetchit>:
4004e0: 31 c0 xor %eax,%eax
4004e2: 0f 18 0d 87 04 20 00 prefetcht0 0x200487(%rip) # 600970 <counter>
4004e9: 48 8b 15 80 04 20 00 mov 0x200480(%rip),%rdx # 600970 <counter>
4004f0: 48 01 c2 add %rax,%rdx
4004f3: 48 83 c0 01 add $0x1,%rax
4004f7: 48 3d 00 84 d7 17 cmp $0x17d78400,%rax
4004fd: 48 89 15 6c 04 20 00 mov %rdx,0x20046c(%rip) # 600970 <counter>
400504: 75 dc jne 4004e2 <prefetchit+0x2>
400506: f3 c3 repz retq
400508: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
40050f: 00
2,445,505,116 cycles # 0.000 GHz ( +- 0.52% )
2,800,857,971 instructions # 1.15 insns per cycle ( +- 0.00% )
0.643010038 seconds time elapsedI think I've discovered that the target of the prefetch is irrelevant, and that what matters is that the store-to-load forwarding does not try to execute in the same cycle. For Sandy Bridge, I'm finding that I can get slightly better performance with a dummy load of an unrelated volatile variable:
volatile unsigned dummy = 0;
void loop_dummy_read() {
IACA_START;
unsigned j;
unsigned dummy_read;
for (j = 0; j < N; ++j) {
dummy_read = dummy;
counter += j;
}
IACA_END;
}That's almost 20% faster than the original. Not bad at all.
The prefetch is only serving as 6-byte padding here. The nopw instruction performs no memory access, it does not double as a prefetch.
There are remarkably few memory accesses actually being performed in that loop. The CPU is using store forwarding to cache the 'memory' accesses in the store buffer, which means most accesses are not even accessing L1 cache (if this were the case, we would not have such a low count of cycles per iteration).
The best result I've got is by inserting a 6-byte nopw right before the store:
.L3:
movq counter(%rip), %rdx
addq %rax, %rdx
addq $1, %rax
cmpq $400000000, %rax
nopw 0x1(%rax,%rax,1)
movq %rdx, counter(%rip)
jne .L3
Here is a version of the code that runs even faster than the asm("call foo") one by inserting nops between the load and the use of the loaded value. However it became even faster by inserting a few nops at the beginning of the loop too.
The fastest version that I could find is to do a prefetch of counter at the end of the loop .... speaking of which isn't that nopw that GCC uses for alignment a prefetch instruction too? Is the CPU tricked by the fake address used there?
vs This is on a Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz