"A tricky beast" definitely seems fair. These things have to scale per-core power consumption from ~13W down to ~3W on the fly to stay within their 280W limit.

To my knowledge, AMD hasn't implemented any proactive measures that are as severe as Intel's AVX512 strategy, where the instructions get split and handled by the narrower vector units for a surprisingly long time while powering up the full-width vector units (and dropping CPU clocks). This AMD instability is only using AVX2: 256-bit SIMD rather than 512-bit. But spread across so many cores, that's still a lot of FPUs to be lighting up.

So basically Intel underclocks itself to do the AVX stuff while AMD underclocks itself ... less?

Core clock speeds are just some of the many variables that need to be adjusted on the fly. Everyone wants to provide as much performance as possible within the platform power limits. But a sudden change in the instruction stream can cause a big swing in a CPU core's power draw even if the core's clock speed and target voltage remain constant. And new instructions can show up a lot faster than the voltage regulators can respond. So handling transients is tricky, but it also looks like this case may be less about transients and more about a particular motherboard not being able to supply stable power for a workload that's tuned to push the limits as much as possible. Gigabyte may have missed their mark when considering what the worst-case power draw situation is that they need to design and test for.

Intel CPUs have two different clock "ladders" so to speak, which define the range of frequencies the dynamic frequency selection (boost) can use. Specifically, as soon as an AVX2 instruction is encountered, the entire thing switches to the AVX2 clock ladder, which drops base and maximum frequencies by a GHz or so. Ladder selection is quite coarse in both cores affected; all cores for smaller CPUs; and time (I believe the y can only change like every 100 ms or so).

Why are the adjustments so coarse here? What's the limit to making them finer, which I assume would make sense?

You have to switch to a higher-voltage state to account for the voltage droop that you will cause when the AVX units fire off.

https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html

(specifically: https://travisdowns.github.io/assets/avxfreq1/fig-volts256-1... )

> So one theory is that this type of transition represents the period between when the CPU has requested a higher voltage (because wider 256-bit instructions imply a larger worst-case current delta event, hence a worst-case voltage drop) and when the higher voltage is delivered. While the core waits for the change to take effect, throttling is in effect in order to reduce the worst-case drop: without throttling33 there is no guarantee that a burst of wide SIMD instructions won’t drop the voltage below the minimum voltage required for safe operation at this frequency.

I assume this is due to AVX2 being rare in general-purpose software, so the design "one core is running AVX2 -> switch large part of CPU to AVX2 mode" doesn't hinder use cases where you are hammering all cores with AVX2 code.

I thought that @Beeonarope had shown that avx2 instructions do not clock down at all. It was only AVX-256 and avx-512 that did.

I believe it depends on which CPU generation you have.

Yeah, I'm referring to the latest generation, of which this AMD should fit that description.

That would suggest the instability might be resolved by adding threads one at a time, but does prime95 have an option like that?

That's something you can do outside of prime95; the schedutil scheduler does that by default, which I believe is why I had hard crashes with ondemand and not schedutil.

(Ok, it actually ramps up frequencies slower rather than lock out cores, but the effect is the same.)

you can probably bodge it by running a bunch of copies with limited workers, then run your main copy, then kill the starters. or, you could fiddle with the CPU affinity.

That's an interesting thought!

I remember when Buildzoid of AHOC did the mobo breakdowns of the TR4 boards and thinking that while some of the super high end boards were probably good for this sort of beating, the mid and low range might struggle hard with the 64/128 part if it ever came into being (3990X was just a rumor at that time). But it looks like they need more caps to handle the transient response time, and probably also some firmware fixes to slow ramp because I don't think all the SMD caps in the world are going to handle that sort of ramp. It's just not possible to get them close enough to the actual CPU without literally putting them under the IHS.

Here's BuildZoid review of the motherboard I'm using (GIGABYTE TRX40 Aorus Xtreme):

https://www.youtube.com/watch?v=HMUWzDSAS9c

A very interesting watch if you're interested in electronics in general and in power delivery in particular (the whole YouTube channel is awesome to be honest).

That was one of the few that I thought could handle the 64/128 part. However look at the output filtering: It's roughly the same if not a cap or two larger as what you'd find on x299... which is a higher voltage and thus has lower amperage requirements. I have yet to find a back of board shot but unless it has a ton of SMD AL-poly caps back there that board would still struggle with the ramp described in the thread. Even then I'm not sure the socket resistance wouldn't cause enough V-droop to cause a crash anyway.

Tangent: is there a reason that CPUs’ instruction sets aren’t designed with explicit “hint”-ops in them to let a compiler assert “I’m going to execute some instructions that’re going to draw a lot of power about 1000 cycles from now, so start ramping up for it now”? That’d basically eliminate what Intel calls “license-switching” costs.

Do they just not believe in compiler authors to be able to emit these kinds of hints? Is there too much legacy code that would come without the hints for it to be worthwhile? Would it just not be worth it given how often OS context switches could drop the CPU directly from regular code in one process to AVX2 code in another process?

There is no explicit "warmup" instruction but you can perform a dummy AVX instruction wherever you want to start warming it up.

> I observed an interesting phenomenon when executing 256-bit vector instructions on the Skylake. There is a warm-up period of approximately 14 µs before it can execute 256-bit vector instructions at full speed. Apparently, the upper 128-bit half of the execution units and data buses is turned off in order to save power when it is not used. As soon as the processor sees a 256-bit instruction it starts to power up the upper half. It can still execute 256-bit instructions during the warm-up period, but it does so by using the lower 128-bit units twice for every 256-bit vector. The result is that the throughput for 256-bit vectors is 4-5 times slower during this warm-up period. If you know in advance that you will need to use 256-bit instructions soon, then you can start the warm-up process by placing a dummy 256-bit instruction at a strategic place in the code. My measurements showed that the upper half of the units is shut down again after 675 µs of inactivity.

https://www.agner.org/optimize/blog/read.php?i=628

Agner also notes that previous generations would just flatly stall all instructions (including non-vector) once they saw 256-bit instructions, which is probably why there is no "warmup" instruction. If there is an explicit stall of all instructions then there are very few reasons to incur that until the last possible second.

I think this would be seen as an overly-leaky abstraction, for several reasons: (1) You don't want the chip's correctness to be at the mercy of the particular instruction stream it's given; (2) the problem is potentially solvable at the microcode level, which makes (1) remain viable.

But, if such "hint" ops were seen as the only viable way to get correct, high-performance behavior in these cases, and there was a commercial case to be made for it, I could see processor vendors adding them to the IA.

Probably because the better approach would be to stall the processor (or at least stall on power intensive instructions) until the requisite power has arrived. 1000 instructions is probably too far out (in terms of instructions) for compilers to accurately add the requisite instructions. Also, it requires code change, unlike the stalling approach. The only disadvantage to the stalling approach would be if the workload is constantly switching between low power and high power, but that can be solved by making the switching interval longer/shorter in the firmware.

Why is it “better” if the stall results in lower performance-per-watt? That’s what people buy these high-end CPU SKUs for, after all. They aren’t just concerned with CapEx (the cost of the CPU) but also OpEx (the aggregate cost of electricity for the CPUs, PSUs and cooling in their studio/data center.) You could also “fix” the problem by just allowing the customer to lock the CPU into AVX2 mode and thereby never switching out at all—but of course that’d be dumb; they’d be drawing more power, and also not going as fast as they could, when executing non-AVX2 code.

Another approach I haven’t seen from either Intel or AMD yet, is to copy ARM’s big.LITTLE architecture: to set up separate “AVX2 cores” that can execute AVX instructions and most basic ALU ops but not e.g. branches, put those on the other side of the die with in-wafer thermal insulation between them and the regular cores; and then throw workloads between regular and AVX2 cores in a way where the AVX2 cores heating up doesn’t mean that the non-AVX2 cores are heating up, and the CPU can go back to full Turbo Boost as soon as the workload is thrown back to the regular cores, because the respective regular core is actually quite cool.

(The performance effects of this are possible to loosely estimate, I think, by writing code that synchronously sets up a GPGPU pass on some data, executes it, retrieves the result, and then returns to executing CPU instructions for a while, in a loop; and then executing this code on an Intel CPU using its on-die IGPU as the GPU. The CPU and IGPU form somewhat-separate thermal domains already, though there’s no explicit insulation.)

Because it doesn't rely on both the user and developers to do some yet to be determined 'right thing' for the CPU to behave in a consistent manner. While you are correct that some people want absolute max performance per watt, I would argue that no one wants a CPU that behaves unpredictably or inconsistently. An option to disable one or more power saving features that result in this issue (assuming that's what the issue is) for those who just need maximum performance could be a desired solution for some. While others, possibly most, would be willing to take a short term performance hit from a stall to maintain the best overall performance per watt characteristics.

> Probably because the better approach would be to stall the processor (or at least stall on power intensive instructions) until the requisite power has arrived.

Sorry to be pedantic, but... physics requires a degree of pedantry.

1. This is probably an ENERGY issue, not a power issue. Energy is the integral of power (or power is the derivative of energy: power is the change of energy over time).

2. Electronics can only really measure voltage easily. Everything else is converted to voltage to be measured. For example, to measure current, you create a resistor and then measure the voltage across the resistor. To measure power, you need to measure voltage, and multiply it with current. To measure energy, you take the integral of your power measurement. (No joke: that's how it typically works).

3. With a capacitor, you can measure your energy level. Capacitors are relatively inaccurate however (+/- 10% tolerances at the electronics level, and probably worse tolerances at the nanometer chipscale level). The voltage from a capacitor does roughly correlate to its energy level.

-----------

Issue #1: Capacitors suck at energy storage, but they're all that engineers have at these speed and sizes.

#2: Voltage is the only thing that can be measured reliably, but it takes a long time (more than a few nanoseconds, aka 10s or 100s of clock cycles) to measure voltage. Measuring other variables, such as "Energy" (voltage level of a capacitor), or "Power" (change of energy over time, or voltage * current), takes even longer to measure.

#3: Computers are fast. 4GHz computer means that you have 0.25 nanoseconds to make a decision per clock tick. Waiting on the results of any calculation on #2 will naturally take dozens or hundreds of clock ticks to accomplish.

#4: That's best case scenario, assuming that capacitors are large enough to actually hold the energy you need to perform your calculations. But in an AVX2 situation, where your CPU suddenly will use up more Power (energy/time) than expected, you need to communicate to your power-circuitry to increase the voltage or current delivery to the chip.

#5: Sending commands to the power-circuits / VRMs, or worse case to the Power Supply, can take many milliseconds (millions of nanoseconds) before the Power Supply responds. A 200W chip at 1.25V will draw 160 Amps of current (0.160 coulomb of electrons per millisecond). A 1 Farad capacitor will lose 0.16 volts per 0.16 coulombs of electrons drawn, and hint-hint, you can't physically fit a 1-Farad capacitor in a chip (https://qph.fs.quoracdn.net/main-qimg-dddab4aa574936f8a61437...).

#6: This means that the only reasonable approach with today's technology, is to wait for the VRMs or Power Supply to respond. Note: The Power Supply will naturally notice the voltage drop and automatically send more electrons down to the motherboard. But the time-scale the PSU operates is on the order of milliseconds. A millisecond is 1-million nanoseconds or ~4-million clock cycles AFTER the AVX2 units started getting used. If you run out of energy before the power gets to your chip... guess what? You get a brownout. Random cells in your chip were now turned off and the state of the CPU is a jumbled mess now.

--------

Where, how, and why to place capacitors around a chip is one of the most difficult PCB-issues an electronic engineer deals with. There's a LOT of "rules of thumb", making PCB-design / capacitors a mystic subject matter for many EEs.

Given the difficulty of the subject of power-delivery, energy calculations, and high-speed signals involved (a chip operates at the GHz band, drawing ~100 Amps on the clock-tick, and then 0 Amps between clock ticks!), I'm almost certain that there's a mistake in the capacitors somewhere.

Its a mystic subject that's very difficult to simulate, especially across varying workloads (AVX vs normal 64-bit code, etc. etc.). That's my bet for what's going wrong here.

Honestly, power-delivery is so awesome and complicated. I'm almost surprised that anything works at all these days.

Stalling is the way to go.

I have my doubts about this being a power issue, because localized power analysis is pretty advanced in IC design and they'd totally have tested for this.

Assuming it is a power issue though, nearly all power is used in dynamic switching, so simply inserting stall cycles will resolve it.

Power rails on silicon chips have quite a lot of capacitance, at least enough for many clock cycles, so you don't even need to make every cycle in every bit of hardware a possible stall cycle - it would be enough to simply gate instruction issue I expect and let functional units drain.

Last time they trusted compilers to become smart Itanium happened.

sTRX4 can take one of three CPUs ranging from 24 to 64 cores. The TRX40 Aorus Xtreme that's mentioned in the thread should be the absolute top of the line even if it was launched before the 64 core monster was available. So I'd expect it to work just fine with the 3970X which is a 32 core part.

But I wonder if this is a Gigabyte issue who have a history of playing around with the power delivery and using "fake phases" (to the point where they now have to advertise their boards as having 16 "real phases"). As far as I can tell many (most?) reviewers benchmarked the board with 24 core CPUs and most likely skipped on the power intensive tests.

To me this smells 100% like a transient issue, Gigabyte probably assumed that more phases would wipe the transient response time out. The issue is that if you're shutting down phases as Buildzoid mentions you'd have to do to get efficiency; you then lose that response time advantage until the phase is spun back up which is the longest part of the power system cycle. Normally output filtering is designed to make up that gap, but I think in the case of these chips that's not happening. I suspect it's a lot more complicated because I have a sneaky suspicion that adding more caps wouldn't solve the problem (completely).

This mobo has 16 real phases, according to BuildZoid's analysis.

I imagine it is. I meant they had such issues in the past (hence the "this time for real" approach) so it may be that they cut different corners in order to meet some other targets that are marketable.

That would be a motherboard issue not a CPU one, though. Has anyone attached an oscilloscope?

Yes: https://forum.level1techs.com/t/3970x-prime95-stability/1532...

Great thread thanks for sharing. Really makes you want to order an oscilloscope and some CPUs to putz around with.

Everything in computer science is such a rabbit hole, great field to be in.

If you're interested in a relatively inexpensive oscilloscope to play around with, check out the Rigol 1054Z (or 1074Z Plus if you want MSO capabilities). It's four channels and has more than enough features for playing around with, especially for the price. Using the Riglol website, you can unlock all software options on it, including increasing the bandwidth to 100mHz and the memory to 24MP.

It was the first 'real' scope I bought and I still use it a fair amount despite having upgraded to a Rohde and Schwarz MSO model. I've been amazed how much I use an oscilloscope after getting one, from measuring ripple on power supplies to diagnosing serial communication issues.

> Using the Riglol website, you can unlock all software options on it, including increasing the bandwidth to 100mHz and the memory to 24MP.

It's interesting how Rigol did the locking of those extra features. The unlock key for a feature set for your scope has to be signed by a Rigol private key using an elliptic curve signature system.

But they are only using a 56 bit private key. That was quickly brute forced, and key generators proliferated.

They used a good library for the cryptography stuff, and except for the short key seem to have used it well and knew what they were doing. This suggests that the choice of a weak key was deliberate.

Each family of scopes has its own private key. As few families came out with new private keys, Rigol continued to use 56 bits. When major firmware upgrades came out in existing families, where they could have easily changed to a longer private key, they kept the same 56 bit that was now widely circulated on the net.

It seems pretty clear that they are not interested in stopping people from free unlocking.

It seems like a decent price discrimination strategy to me. They make advanced capabilities more price-accessible to particularly interested hobbyists and more popular among that market, and probably aren't losing much revenue from corporate and academic institution sales.

Thanks for the recommendations! I have a DSLabs DScope (100 MHz, 2-channel FPGA scope) and while it’s handy I’d prefer to have a proper hardware scope someday. Rigol’s scopes look like they nicely fit in between the basic DSO/FPGA stuff and the “proper” 4-5 digit priced test bench gear.

Any recommendations for learning resources that could help with understanding DC power supply analysis for non-EE types? While refurbishing laptops and working with microcontrollers I’ve run into some odd things where ruling out transient power supply issues would probably be helpful.

The low end Rigols make good entry-level scopes, and have a surprising amount of capability for the price.

As for learning resources, I came across a decent article on the subject when I was starting out (1), and most of the oscilloscope manufacturers have whitepapers on SMPS diagnostics, the Tektronics one I read a while back (2) gave a good overview. A lot of the whitepapers have a manufacturer-specific focus, but they still have good information that can be applied to almost any oscilloscope.

If you want to get really into the power supply and do high-side measurements you'll need an isolated differential probe, which can cost as much as an inexpensive oscilloscope, but for DC output measurement you shouldn't need anything special. Current probes are a lot more affordable if you're interested in looking at loads or current fluctuations/harmonics, but that's more useful after you've figured out a bit more what specific properties you're trying to measure.

1: https://www.testandmeasurementtips.com/test-switching-power-...

2: https://download.tek.com/document/3GW_23612_7.pdf

Edit: I forgot to mention that the EEVblog forums are a good resource also, but they sometimes aren't as friendly as they could be towards people just starting out.

Wow neat, thanks! Happy to have helped spawn this little thread here. Will definitely check out Rigol.

I'm really confused the OP there didn't try a different sTRX4 motherboard.

OP here. I wish I had! Unfortunately that's my work machine and I don't have the time to somehow get my hands on a new fairly rare, very expensive motherboard, disassemble my workstation and rebuild it just to see if my motherboard is the culprit. From the experience of other owners of the 3970X, this problem happens with other TRX40 motherboards.

At least EU Amazon is extremely good about RMAs, e.g. international expresses delivery (next day) even without returning the defective component 1st. Return shipment is also covered, i.e. free for the customer. Dunno, if they'd do that for high priced stuff, though.

p.s. lovely/lively oscilloscope shots!

Makes sense. I’ve done Amazon Prime and returned it because I consider that a defective mobo and fair game to return in the past.

> fortunately fixable if you disable all spread-spectrum

If these are on by default, are they required for FCC certification conformance? Presumably the device has not been EMI/EMC tested in the mode where spread-spectrum was disabled.

They are, but I'm not physically in the USA. At any rate, isn't a computer case basically a Faraday cage?

The frequencies are up in the microwave band, so I can't imagine there's much range.

> computer case basically a Faraday cage?

If it was designed well ;)

But that's only effective for addressing radiated emissions, not conducted ones.