News for hackers. Stuff that matters.
Re:63 Cores Blocked by Seven Instructions by saagarjha on Monday October 21, 2019 @04:53AM (#21310361)
Why do the sample counts cluster so heavily on the jne, as opposed to the other instructions in the loop?
Re:63 Cores Blocked by Seven Instructions by jclulow on Monday October 21, 2019 @05:03AM (#21310406)
The article links to another article which discusses this. In order to do stack sampling, the CPU doing the work must be periodically interrupted for the sampler to run and collect a stack. Modern CPUs are deeply speculative and often issue instructions out of strict program order. When you interrupt such a CPU it has to walk some of that back and decide where to leave off on all of the speculative execution; what the CPU was doing at the time is thus not completely captured by the stack trace alone.
Re:63 Cores Blocked by Seven Instructions by brucedawson on Monday October 21, 2019 @11:41PM (#21320201)
I talked to the author of that article. He hasn't done testing on AMD processors but his guess was:

micro-up fusion means the seven-instruction loop is actually five micro-ops Zen2 processors can retire five instructions per cycle Therefore the loop runs at one iteration per cycle (wow!) The cmp [r8] instruction occasionally has cache misses This means that the seven instructions get synchronized such that the cmp [r8] instruction is the last of them to get retired in a seven-instruction block Therefore the next instruction is usually the jne

TL;DR - the jne gets most of the samples because the cmp [r8] instruction is the most expensive.

Re:63 Cores Blocked by Seven Instructions by BeeOnRope on Tuesday October 22, 2019 @01:49AM (#21320698)
Yeah I believe jne gets most of the samples because cmp [r8] is the most expensive, but there could be two separate reasons for that:

Perhaps ETW shows you the precise instruction (i.e., "zero skid") that is slow to retire - this is not like a normal interrupt as described in the article but is available with some performance profiling events like 'cycles:ppp' on Linux perf (in particular, using the zero-skid PEBS events).

In that case, the samples shows up on the jne, not the cmp likely because cmp/jne have fused, so basically get sampled as a single instruction and the samples show up pointing to the jump.

The other scenario is that ETW shows you "skid 1" instructions, i.e., the instructions generally after the slow-to-retire ones (as described in the article), and cmp/jne didn't fuse (perhaps because a cmp with a memory source argument can't fuse on AMD?), and so it again points to the jne.

I haven't looked at many ETW traces, so I couldn't tell you offhand - but for those who have, do the samples usually show on the expensive instructions (things like div and loads that miss are a giveaway), or on the one after?

Added: Per Agner, I guess the "fusion + no skid" is the most likely (from the Ryzen section of microarchitecture.pdf):

> A CMP or TEST instruction immediately followed by a conditional jump can be fused into a single μop. This applies to all versions of the CMP and TEST instructions and all conditional jumps, except if the CMP or TEST instruction has a rip-relative address or both a displace- ment and an immediate operand.

That also lines up with the cmp having exactly 0 samples, unlike any other instruction of the 7: that's a common indication of fusion.

Re:63 Cores Blocked by Seven Instructions by Const-me on Monday October 21, 2019 @06:38AM (#21310749)
It's the same off-by-one issue that often affects address of crashing instruction.

The report is incorrect, vast majority of time is taken by the previous one, cmp dword ptr [r8],ebp. It's the only one accessing RAM, and accessing cache line shared across cored is very expensive, even more so than a cache miss.

Re:63 Cores Blocked by Seven Instructions by brucedawson on Monday October 21, 2019 @11:36AM (#21313480)
What off-by-one on crashing instructions are you referring to? Some stack walkers intentionally do off-by-one byte offsets when doing symbol/source lookup on parent functions, but not on the crashing instruction.

Crashing is (with the exception of floating-point exceptions) precise. A particular instruction crashes, the exception record points there, the instructions afterwards are discarded with no side effects. This is necessary to support things like restarting execution.

Sampling, on the other hand, is not even well defined. There are hundreds of instructions in flight, many completing simultaneously, and when an external interrupt happens the CPU has to decide which ones to commit and which to discard. The linked article gives many more thoughts about how the CPU draws the line in the silicon.

Re:63 Cores Blocked by Seven Instructions by Const-me on Monday October 21, 2019 @04:06PM (#21316678)
> What off-by-one on crashing instructions are you referring to?

Here’s an example for GCC on ARM Linux: https://github.com/dotnet/corert/issues/7826 I think I have observed similar symptoms on Windows, too.

> Sampling, on the other hand, is not even well defined

A crash by e.g. RAM access violation, and interrupt generated by CPU to collect sample for a profiler, are pretty similar, IMO.

Re:63 Cores Blocked by Seven Instructions by rrss on Monday October 21, 2019 @11:33PM (#21320168)
access violations are generated internally by a single instruction, and all mainstream CPUs guarantee precise exceptions. PMU interrupts for sampling are external so the CPU picks wherever it wants to stop in the program.
Re:63 Cores Blocked by Seven Instructions by Const-me on Monday October 21, 2019 @11:53PM (#21320231)
> all mainstream CPUs guarantee precise exceptions

Yes, and precise interrupts, too.

> CPU picks wherever it wants to stop in the program.

I've used profilers quite a lot, and based on my observations they're quite accurate, to exact instruction.

Re:63 Cores Blocked by Seven Instructions by rrss on Tuesday October 22, 2019 @11:17AM (#21324589)
> Interrupt-based sampling introduces skids on modern processors. That means that the instruction pointer stored in each sample designates the place where the program was interrupted to process the PMU interrupt, not the place where the counter actually overflows, i.e., where it was at the end of the sampling period. In some case, the distance between those two points may be several dozen instructions or more if there were taken branches.

https://perf.wiki.kernel.org/index.php/Tutorial#Event-based_...

Re:63 Cores Blocked by Seven Instructions by brucedawson on Wednesday October 23, 2019 @03:01AM (#21331883)
But what does that even mean? Seriously. An interrupt fires, on a particular clock tick. At that point there are, let's say, 130 instructions in flight. In the case of a loop like this one there may be seven instructions being retired per clock-cycle.

So, you end up with patterns. I linked to some detailed reverse engineering of which instructions are likely to end up being the victim. One common pattern is that the instruction after an expensive one will have the samples assigned to it, but there's more to it than that - I recommend reading it.

TL;DR - I'm not saying you're wrong, it's just that you're not saying anything specific enough for write/wrong to apply. "accurate, to exact instruction" has not been meaningful for sampling profilers for more than 2.5 decades.

Re:63 Cores Blocked by Seven Instructions by BeeOnRope on Tuesday October 22, 2019 @01:53AM (#21320718)
I don't think that's the case here - see my other longer comment, but I think ETW uses the PEBS sampling events, so the instruction is usually the slow-to-retire one, not the subsequent one.

I believe cmp/jne macro fuse on this CPU, so you actually will never get any samples on the first of the two fused instructions: rather they all show on the second one. You see this same effect on Linux when sampling with the cycles:ppp event.

Re:63 Cores Blocked by Seven Instructions by brucedawson on Wednesday October 23, 2019 @03:07AM (#21331928)
I have seen no signs of samples hitting the most expensive instruction in ETW trace. Across other cases I have looked at the samples tend to cluster after expensive instructions, not on them.

I guess it's possible that all of the cases I looked at were distorted by macro fusion but I don't think so.

Re:63 Cores Blocked by Seven Instructions by BeeOnRope on Wednesday October 23, 2019 @11:41AM (#21335756)
Interesting, so I guess ETW is using "normal" interrupts which have skid, and the CPU just having enough retire bandwidth to retire everything in one cycle.
Re:63 Cores Blocked by Seven Instructions by Const-me on Tuesday October 22, 2019 @09:37AM (#21323426)
> macro fuse on this CPU, so you actually will never get any samples on the first of the two fused instructions

Haven’t thought about µ-ops fusion in this case. Yes, that explanation is very plausible.