Here's BuildZoid review of the motherboard I'm using (GIGABYTE TRX40 Aorus Xtreme):

https://www.youtube.com/watch?v=HMUWzDSAS9c

A very interesting watch if you're interested in electronics in general and in power delivery in particular (the whole YouTube channel is awesome to be honest).

That was one of the few that I thought could handle the 64/128 part. However look at the output filtering: It's roughly the same if not a cap or two larger as what you'd find on x299... which is a higher voltage and thus has lower amperage requirements. I have yet to find a back of board shot but unless it has a ton of SMD AL-poly caps back there that board would still struggle with the ramp described in the thread. Even then I'm not sure the socket resistance wouldn't cause enough V-droop to cause a crash anyway.

Tangent: is there a reason that CPUs’ instruction sets aren’t designed with explicit “hint”-ops in them to let a compiler assert “I’m going to execute some instructions that’re going to draw a lot of power about 1000 cycles from now, so start ramping up for it now”? That’d basically eliminate what Intel calls “license-switching” costs.

Do they just not believe in compiler authors to be able to emit these kinds of hints? Is there too much legacy code that would come without the hints for it to be worthwhile? Would it just not be worth it given how often OS context switches could drop the CPU directly from regular code in one process to AVX2 code in another process?

There is no explicit "warmup" instruction but you can perform a dummy AVX instruction wherever you want to start warming it up.

> I observed an interesting phenomenon when executing 256-bit vector instructions on the Skylake. There is a warm-up period of approximately 14 µs before it can execute 256-bit vector instructions at full speed. Apparently, the upper 128-bit half of the execution units and data buses is turned off in order to save power when it is not used. As soon as the processor sees a 256-bit instruction it starts to power up the upper half. It can still execute 256-bit instructions during the warm-up period, but it does so by using the lower 128-bit units twice for every 256-bit vector. The result is that the throughput for 256-bit vectors is 4-5 times slower during this warm-up period. If you know in advance that you will need to use 256-bit instructions soon, then you can start the warm-up process by placing a dummy 256-bit instruction at a strategic place in the code. My measurements showed that the upper half of the units is shut down again after 675 µs of inactivity.

https://www.agner.org/optimize/blog/read.php?i=628

Agner also notes that previous generations would just flatly stall all instructions (including non-vector) once they saw 256-bit instructions, which is probably why there is no "warmup" instruction. If there is an explicit stall of all instructions then there are very few reasons to incur that until the last possible second.

I think this would be seen as an overly-leaky abstraction, for several reasons: (1) You don't want the chip's correctness to be at the mercy of the particular instruction stream it's given; (2) the problem is potentially solvable at the microcode level, which makes (1) remain viable.

But, if such "hint" ops were seen as the only viable way to get correct, high-performance behavior in these cases, and there was a commercial case to be made for it, I could see processor vendors adding them to the IA.

Probably because the better approach would be to stall the processor (or at least stall on power intensive instructions) until the requisite power has arrived. 1000 instructions is probably too far out (in terms of instructions) for compilers to accurately add the requisite instructions. Also, it requires code change, unlike the stalling approach. The only disadvantage to the stalling approach would be if the workload is constantly switching between low power and high power, but that can be solved by making the switching interval longer/shorter in the firmware.

Why is it “better” if the stall results in lower performance-per-watt? That’s what people buy these high-end CPU SKUs for, after all. They aren’t just concerned with CapEx (the cost of the CPU) but also OpEx (the aggregate cost of electricity for the CPUs, PSUs and cooling in their studio/data center.) You could also “fix” the problem by just allowing the customer to lock the CPU into AVX2 mode and thereby never switching out at all—but of course that’d be dumb; they’d be drawing more power, and also not going as fast as they could, when executing non-AVX2 code.

Another approach I haven’t seen from either Intel or AMD yet, is to copy ARM’s big.LITTLE architecture: to set up separate “AVX2 cores” that can execute AVX instructions and most basic ALU ops but not e.g. branches, put those on the other side of the die with in-wafer thermal insulation between them and the regular cores; and then throw workloads between regular and AVX2 cores in a way where the AVX2 cores heating up doesn’t mean that the non-AVX2 cores are heating up, and the CPU can go back to full Turbo Boost as soon as the workload is thrown back to the regular cores, because the respective regular core is actually quite cool.

(The performance effects of this are possible to loosely estimate, I think, by writing code that synchronously sets up a GPGPU pass on some data, executes it, retrieves the result, and then returns to executing CPU instructions for a while, in a loop; and then executing this code on an Intel CPU using its on-die IGPU as the GPU. The CPU and IGPU form somewhat-separate thermal domains already, though there’s no explicit insulation.)

Because it doesn't rely on both the user and developers to do some yet to be determined 'right thing' for the CPU to behave in a consistent manner. While you are correct that some people want absolute max performance per watt, I would argue that no one wants a CPU that behaves unpredictably or inconsistently. An option to disable one or more power saving features that result in this issue (assuming that's what the issue is) for those who just need maximum performance could be a desired solution for some. While others, possibly most, would be willing to take a short term performance hit from a stall to maintain the best overall performance per watt characteristics.

> Probably because the better approach would be to stall the processor (or at least stall on power intensive instructions) until the requisite power has arrived.

Sorry to be pedantic, but... physics requires a degree of pedantry.

1. This is probably an ENERGY issue, not a power issue. Energy is the integral of power (or power is the derivative of energy: power is the change of energy over time).

2. Electronics can only really measure voltage easily. Everything else is converted to voltage to be measured. For example, to measure current, you create a resistor and then measure the voltage across the resistor. To measure power, you need to measure voltage, and multiply it with current. To measure energy, you take the integral of your power measurement. (No joke: that's how it typically works).

3. With a capacitor, you can measure your energy level. Capacitors are relatively inaccurate however (+/- 10% tolerances at the electronics level, and probably worse tolerances at the nanometer chipscale level). The voltage from a capacitor does roughly correlate to its energy level.

-----------

Issue #1: Capacitors suck at energy storage, but they're all that engineers have at these speed and sizes.

#2: Voltage is the only thing that can be measured reliably, but it takes a long time (more than a few nanoseconds, aka 10s or 100s of clock cycles) to measure voltage. Measuring other variables, such as "Energy" (voltage level of a capacitor), or "Power" (change of energy over time, or voltage * current), takes even longer to measure.

#3: Computers are fast. 4GHz computer means that you have 0.25 nanoseconds to make a decision per clock tick. Waiting on the results of any calculation on #2 will naturally take dozens or hundreds of clock ticks to accomplish.

#4: That's best case scenario, assuming that capacitors are large enough to actually hold the energy you need to perform your calculations. But in an AVX2 situation, where your CPU suddenly will use up more Power (energy/time) than expected, you need to communicate to your power-circuitry to increase the voltage or current delivery to the chip.

#5: Sending commands to the power-circuits / VRMs, or worse case to the Power Supply, can take many milliseconds (millions of nanoseconds) before the Power Supply responds. A 200W chip at 1.25V will draw 160 Amps of current (0.160 coulomb of electrons per millisecond). A 1 Farad capacitor will lose 0.16 volts per 0.16 coulombs of electrons drawn, and hint-hint, you can't physically fit a 1-Farad capacitor in a chip (https://qph.fs.quoracdn.net/main-qimg-dddab4aa574936f8a61437...).

#6: This means that the only reasonable approach with today's technology, is to wait for the VRMs or Power Supply to respond. Note: The Power Supply will naturally notice the voltage drop and automatically send more electrons down to the motherboard. But the time-scale the PSU operates is on the order of milliseconds. A millisecond is 1-million nanoseconds or ~4-million clock cycles AFTER the AVX2 units started getting used. If you run out of energy before the power gets to your chip... guess what? You get a brownout. Random cells in your chip were now turned off and the state of the CPU is a jumbled mess now.

--------

Where, how, and why to place capacitors around a chip is one of the most difficult PCB-issues an electronic engineer deals with. There's a LOT of "rules of thumb", making PCB-design / capacitors a mystic subject matter for many EEs.

Given the difficulty of the subject of power-delivery, energy calculations, and high-speed signals involved (a chip operates at the GHz band, drawing ~100 Amps on the clock-tick, and then 0 Amps between clock ticks!), I'm almost certain that there's a mistake in the capacitors somewhere.

Its a mystic subject that's very difficult to simulate, especially across varying workloads (AVX vs normal 64-bit code, etc. etc.). That's my bet for what's going wrong here.

Honestly, power-delivery is so awesome and complicated. I'm almost surprised that anything works at all these days.

Stalling is the way to go.

I have my doubts about this being a power issue, because localized power analysis is pretty advanced in IC design and they'd totally have tested for this.

Assuming it is a power issue though, nearly all power is used in dynamic switching, so simply inserting stall cycles will resolve it.

Power rails on silicon chips have quite a lot of capacitance, at least enough for many clock cycles, so you don't even need to make every cycle in every bit of hardware a possible stall cycle - it would be enough to simply gate instruction issue I expect and let functional units drain.

Last time they trusted compilers to become smart Itanium happened.

sTRX4 can take one of three CPUs ranging from 24 to 64 cores. The TRX40 Aorus Xtreme that's mentioned in the thread should be the absolute top of the line even if it was launched before the 64 core monster was available. So I'd expect it to work just fine with the 3970X which is a 32 core part.

But I wonder if this is a Gigabyte issue who have a history of playing around with the power delivery and using "fake phases" (to the point where they now have to advertise their boards as having 16 "real phases"). As far as I can tell many (most?) reviewers benchmarked the board with 24 core CPUs and most likely skipped on the power intensive tests.

To me this smells 100% like a transient issue, Gigabyte probably assumed that more phases would wipe the transient response time out. The issue is that if you're shutting down phases as Buildzoid mentions you'd have to do to get efficiency; you then lose that response time advantage until the phase is spun back up which is the longest part of the power system cycle. Normally output filtering is designed to make up that gap, but I think in the case of these chips that's not happening. I suspect it's a lot more complicated because I have a sneaky suspicion that adding more caps wouldn't solve the problem (completely).

This mobo has 16 real phases, according to BuildZoid's analysis.

I imagine it is. I meant they had such issues in the past (hence the "this time for real" approach) so it may be that they cut different corners in order to meet some other targets that are marketable.