I debugged this problem over a number of years... I replaced the RAM, I replaced the motherboard, I eventually replaced the CPU... still happened no matter what I did. Even if I underclocked the machine and kept the voltages the same, instant crash. It was maddening.
Exasperated, I eventually busted out an oscilloscope and looked at the waveform on the 12V supply to the CPU. When starting the AVX benchmark, there was a huge brownout. That basically explained everything; my power supply essentially turned off when the CPU started drawing a ton of power. I replaced the power supply and got lucky -- it handled it fine and I could run the benchmark. I even got some overclock out of it.
After this whole experience, I've never looked at computers the same way again. You can buy high-spec brand-name components, and it's all just a crapshoot. Maybe your computer won't crash in the middle of an important task. Maybe it will. There isn't much you can do but cross your fingers.
I overbuy power supplies because I think, but have not tested, that it will convince them to run the fan at lower speed. This strategy has resulted in quiet, for sure, but I'm not sure that it gets you reliability.
This theory is sound. High-end power supplies tend to not turn on the fan until reaching 30-50% of their rated maximum output. Fanless power supplies are usually internally using the same components as regular high-end models with 80-100% higher output ratings, but with larger heatsinks. A high-quality oversized power supply with a fan will be silent most of the time, but keep itself cool when necessary.
( I have a semi-passive 750W power supply that has spun up its fan exactly once, when I accidentally shorted two of its power rails together and melted some wires while burning out the PCIe slot I was measuring the power draw through. The motherboard and wires had to be replaced, but the CPU, PSU and PCIe card were all fine.)
Spending days trying to chase down an intermittent reboot just to find that your 750W PSU is actually shitting the bed around ~350W is a terribly unsatisfying journey.
These days I don't know if that's entirely true, so I would at least look at reviews to make sure.
Anecdotally, Corsair has always been a top-notch company to me. Even when their products go bad, their support is awesome. I had a set of memory sticks appear to go bad, but I couldn't tell if it was my northbridge or the RAM itself. I was a cash-strapped kid, so I didn't have another machine to test with. Corsair sent out a set of known-good RAM sticks for me to test with, at completely no charge to me. When the test revealed that it was indeed the RAM that was going bad, they offered to let me keep the test sticks they sent me until my replacement came in the mail.
The OEM for most of their PSUs is CWT, but they are "Corsair custom" designs instead of off-the-shelf. Seasonic is only used for relatively few of Corsair's models.
Only one out of ~ 10 I bought ever failed on me, and that was after 2 years of no usage.
Jonnyguru's last Seasonic review: https://www.jonnyguru.com/blog/2018/07/03/seasonic-prime-ult...
Literally nothing in the bad & mediocre summary sections, and with nothing but recommendations for the entire series:
> Our look at the PRIME Ultra Titanium series has now at last come to a conclusion. We can now definitively say there’s not a bad performer in the bunch, and while all do have some very minor drawbacks I can’t see any reason not to recommend any of them.
Test the RAM in a known stable machine. If it passes an overnight memtest, then replace the flaky machine's PSU.
I avoid XMP, however, and typically manually specify the voltage that is listed in the RAM's spec sheet and run at the speed that the CPU manufacturer recommends. (So the RAM might be sold as 3200 but Intel says 2400 is the max, so I run it at 2400.)
[1] https://www.anandtech.com/show/14525/amd-zen-2-microarchitec...
You need 3600MT to match that while keeping 1:1 sync, which means XMP unfortunately.
Gaming hardware has larger economies of scale which leads to more people running the hardware and submitting RMAs. Gamers are also more likely to overvolt their hardware, leading to most consumer components being overspec'ed for heat and power.
As such, the second revision of a gaming board imo is more refined than a workstation/server board that almost never gets a new revision.
On a more practical level, consumer CPUs usually have higher clockrates and newer architectures and gaming NVidia GPUs are so much cheaper than the enterprise product lines that you can buy multiple. My workflow is faster with the cheap stuff, sometimes twice as fast.
I think you're also overestimating how big the gaming desktop components market is, and underestimating how fragmented it is. For motherboards, the economies of scale apply to the OEM custom motherboards used by Dell, HP, etc., which have far more volume than any one consumer retail motherboard model. You're probably also vastly overestimating how many gaming-oriented motherboards make it past revision 1.0. I just checked Gigabyte's top gaming product line, and out of 75 models listed, all but 8 were revision 1.0. Browsing through a similar number of their most recent mainstream consumer motherboards didn't turn up any that went beyond rev 1.0.
Some gamer-oriented motherboards are over-specced for power delivery and cooling. It's definitely common to find VRM heatsinks that are oversized gaudy sculptures, but that doesn't help product reliability. There are also tons of examples of gamer-oriented boards that advertise a large number of VRM phases, but have obvious shortcomings in their power delivery design once the heatsinks are removed and an EE takes a close look at things. And then there's the motherboard at issue here. When gamer boards are overbuilt, it's not thorough, it's usually just the most visible aspects that get beefed up (and the price).
RAM is a really obvious area where paying a bit extra for workstation-class ECC DIMMs gets you a real reliability advantage, but paying extra for high-speed consumer modules with RGB LEDs and big heatsinks does not.
RGB LEDs are still thankfully pretty uncommon on SSDs, but the handful of examples show that the impact varies from compromised performance to catastrophic instability. Most gamer-oriented SSDs with no LEDs but with flashy heatsinks also fail to outperform or outlive the best pedestrian-looking models.
Enterprise and NAS hard drives are definitely diverging from mainstream consumer hard drives in more ways than simple firmware feature gating. Some of the technologies they're using to drive density beyond consumer HDD needs have a questionable impact on reliability, but filling them with helium does help reliability, and you don't find that on a WD Black or Seagate Barracuda.
"Using AllFrame™ technology, WD Purple™ drives improve video capturing and help to reduce errors, pixelation, and video interruptions that can occur in a video recorder system."
"WD Purple 8TB, 10TB, 12TB & 14TB capacities feature AllFrame AI technology that enables not only recording up to 64 cameras, but also supports an additional 32 streams for Deep Learning analytics within the system."
"WD Purple drives with AllFrame AI technology feature a workload rating up to 360TB/year to support the Deep Learning analytics that is featured in AI capable NVRs."
https://documents.westerndigital.com/content/dam/doc-library...
https://rog.asus.com/articles/usa/rog-overclocking-guide-cor...
It also does not test all parts of the processor equally, it really just is slamming the cache and the AVX units. You can be "Prime95 stable" and still crash in other things. It's just not a good test anymore in multiple respects.
Same thing for Furmark on GPUs. And nowadays GPUs are actually smart enough to realize they're running a power virus and the power management pulls back the power. So again, it doesn't demonstrate anything. Zen2 is supposed to work the same way, if it realizes you're redlining the chip then it should be clocking down somewhat.
On Intel you can also intentionally "triple fault" the CPU and that'll cause it to reset. The way that works is you tell the CPU that it doesn't have interrupt handlers anymore, then cause a software interrupt. Then the CPU will first try to call your interrupt handler (but that'll fail because you told it there are no handlers), then errors during an interrupt cause a "double fault" interrupt that's supposed to handle those errors (but once again, there's no handlers), so now you're in a "triple fault" condition and the only way to recover is to reset, so that's what the CPU does.
Would you buy a GPU that you could safely run Furmark on 24/7 if it was limited to, half or 3/4 of the clocks? That's basically what you're proposing. I seriously doubt that you would actually lay money down for that shit. The market as a whole certainly wouldn't, they would buy the card that's faster under actual real-world workloads and/or didn't require manual overclocking to achieve its potential in real-world workloads.
The good news is that you can have it both ways. Modern power management is smart enough to figure out when you're running a power virus and it will downclock to those safe levels, while us normies can get competitive clocks when running our real-world code. It works pretty well most of the time.
Given that modern GPUs have complex frequency scaling algorithms that take temperature and power usage into account when setting clock speed, there's a good chance that a modern GPU will do exactly that.
> Modern power management is smart enough to figure out when you're running a power virus and it will downclock to those safe levels, while us normies can get competitive clocks when running our real-world code.
Which is perfectly fine, as long as the clock speed behavior is disclosed (which it is).
What isn't fine is a product running at such a high speed from the factory that running certain workloads damages it.
I believe that's where the issue lays. Overclocking usually also means increasing the voltage.
So what you are really saying is Intel made another defective product.
AMD is subject to electromigration as well. I've seen a couple guys on Reddit already degrade their Zen2 chips from pushing 1.4V through them just during normal usage, not even Prime95. Literally weeks of usage.
Anti intel circlejerk ahoy.
Nice contrarianism.
Yes, you can run it, and it should run successfully. Yes, sustained testing will rapidly damage your processor and cause notable performance degradation (reduced maximum overclocks/increase in required voltages) within a matter of days, and yes, you can still crash in Aida64 or other stress tests because it's not testing the whole processor very well.
It's far outside the normal current draw, even compared to other AVX2 loads like video encoding. It's an AVX load that fits entirely inside instruction cache and will hammer the AVX2 units nonstop at the maximum possible rate with absolutely no other instructions going to any of the other units. That's not how any other workload behaves, even video encoding. CPUs are not designed around that, they're designed for some reasonably normal mix of instructions.
Yes, it's a good idea to stress your processor a bit more than normal to prove it's stable, but if you make your 'proof charge' too large you will still damage the firearm.
Furthermore, it should be able to do this 24/7 for years at a time.
This test, assuming it actually does kill the processor or stops executing, clearly indicates it should be returned.
Overclocking? Yeah, it'll kill it quick, and that's your problem, because you're operating it outside normal specifications (more voltage than designed). If that's your standard, that you should be able to overclock and then redline the chip 24/7 for an unlimited period of time and it should never kill the chip, then you're going to be disappointed.
The fact that Threadripper is having problems is definitely a fault but it's something that you really shouldn't be doing anyway (and it sounds like the problem is the motherboard not the processor).
More generally: this is like arguing that you should be able to buy a camry and run redlined 24/7 and get the inevitable engine failures covered by warranty. If the manufacturer finds out you're doing that, they're not going to cover it, because it's not normal usage. Somehow people think it's a reasonable expectation for processors.
This is like the cooling argument all over again - "your cooling system is good enough for normal workloads". No, it must be good enough for permanent full load! Like a CPU under Prime95. If your heatsink lets it overheat, it's defective. You saved a few pennies on the copper and now it's a defective product.
Same for Prime95, the CPU must be able to run it for days at end and last ~5 years. Doesn't seem like too much to ask for, imo. Intel says their chips will run at 98 degrees for years, so why would a full load like Prime95 kill them in days?
One other thing, overclocking these days is actually undervolting combined with raising clocks. The old school "just raise the voltage" is a bad way to do it on any Intel chips post-Sandy Bridge...
If you overclock, you're responsible. Losing warranty sounds acceptable to me. Better than manufacturers doing everything to make overclocking impossible.
To me, it's a great way to get more performance out of an old chip, the warranty would be expired at that point.
It is hard to prove a CPU was overclocked though, so I can see some people taking advantage of that.
Nowadays it's just "up to XX GHz", and people expect it to run at those clocks.
AVX really pushes the CPU, an Intel Core/Xeon will hit the power draw limit pretty fast. They don't decrease clock speed as much as fall back to base clocks. The ones you paid for. Anything above that is just a bonus.
That said, you should never trigger PROCHOT even under full stress load with AVX! If you do, you need better cooling.
It's a last resort throttling feature for when your processor is hot enough to boil water :/
I still don’t understand what it is about avx2 that results in these kinds of issues - is it really just a matter of increased number of execution units running at once causing weird power and heat issues?
That's been the root cause of all the previous AVX weirdness I'm aware of. It's probably the case for these Threadrippers, too. Modern CPUs have very high power density and run at pretty low voltage, which results in insane current delivery requirements.
A normal multiply does 1 number at a time (32 bit for simplicity). AVX2 can do up to 8 multiplications at a time. That's a huge amount more transistors firing all at once and that causes the voltage to start to droop.
AVX-512 takes that even further and now it's 16 multiplications per unit, oh and Intel moved from 1x256-bit unit per core on Haswell/Broadwell to 2x512-bit units on Skylake-X, so it's potentially 32 multiplications at a time - 4x as much as AVX2.
Basically to prep for that much power being drawn all at once, the chip has to switch to a higher-voltage mode to account for the voltage droop caused by all those transistors switching at once in one place. It takes time for the regulator (on-chip Fully Integrated Voltage Regulator or motherboard VRM) to wind up the voltage far enough.
https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html
https://travisdowns.github.io/assets/avxfreq1/fig-volts256-1...
At this level behavior is intensively analog and thermals/voltage both significantly affect transistor current draw and switching time, which feeds back into thermals and power consumption/voltage droop.
This gets even more problematic on 7nm/10nm type nodes and especially in GPUs where you are doing a huge amount of vector arithmetic all the time. Essentially it is no longer possible to design processors that are 100% stable under all potential execution conditions, or even under normal operating conditions, so you have to have power watchdog circuitry that realizes when it's getting close to brownout/missing its timing conditions and slows itself down to stay stable. That's why AMD introduced clock stretching in a big way with Zen2 (despite the fact that it's nominally been around since Steamroller). NVIDIA piloted this with Pascal, AMD piloted it with Vega and brought it to CPU with Zen2. You simply cannot design the processor to be 100% stable at competitive clocks anymore. You have to have power management that's smart enough to withstand small transient power conditioning faults.
https://semiengineering.com/managing-voltage-drop-at-107nm/
https://semiengineering.com/power-delivery-affecting-perform...
Most (all?) Intel server CPUs in fact decrease clock speed when executing AVX2 (and some other) instructions to keep things a bit more sane. Vlad from Cloudflare wrote about this, more specific to AVX-512 back in 2017: https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...
Then there is PROCHOT signal. Which is supposed to protect the CPU from getting too hot but keeps getting raised in lopsided AVX2 loads not because CPU is too hot but voltage regulation gets whacked.
You may wonder: what is an example of AVX2 heavy load. RSA multiplication is a good candidate. AES constructions or modes (CBC with SHA, GCM) are implemented in AVX2-BMI2 as well.