I disagree. This code is just lucky enough to be able to detect the data corruption, but without very deep understanding afterward the whole system state has to be assumed to be corrupted. You can retry until the code doesn't detect data corruption, but you have to assume other state is also corrupted. The right thing would be to scream loudly at the user that their system is unstable and to expect (future) data corruption unless the hardware + firmware (or its configuration) is fixed.

Sure it's unpleasant to be the messenger of bad news, but the alternatives are far worse unless the system is just a dedicated game console without any background processes (which isn't how those CPUs are used).

Yeah, I had a RAM issue that might or might not have involved turning on XMP, with eventually as a result several RAM sticks with errors (BadRAM is amazing BTW, sadly, *nix-only), and worse, corrupted storage partitions !

Was a real pain to deal with the fallout...

As noted in the linked page, this issue would affect any heavy use of CPU. Oodle happened to be optimized well to hit this issue earlier than most other applications, but nothing can't be really trusted at that point. There is a reason that they recommend to disable overclocking if possible, because such issue is in general linked to the instability due to excessive overclocking.

Retry is not an unusual response to unreliable hardware, and all hardware is ultimately unreliable.

Software running at scale in the cloud is written to be resilient to errors of this nature; jobs are cattle, if jobs get stuck or fail they are retried, and sometimes duplicate jobs are started concurrently to finish the entire batch earlier.

Cloud machines have a way better guarantee about such errors though. You will eventually see some errors at scale, but that error rate can be reliably quantified and handled accordingly.

Consumer machines are comparably wild. Remember that this issue was mainly spotted from Unreal error messages. Some do too much overclocking without enough testing, which will eventually harm the hardware anyway. Some happen to live in places where single-error upsets are more frequent (for example, high altitude or more radioactive bedrock). Some have an insufficient power supply that causes erratic behaviors only on heavy load. All those conditions can be handled in principle, but are much harder to do so in practice. So giving up is much more reasonable in this context.

That same hardware bug will also result in other corruption in other places where it's not detected, which may then spiral into much more catastophic behaviour. Oodle is just special in that it detects the corruption and throws an error (which is the right thing to do in this situation IMHO).

The ball is in Intel's court, such faulty CPUs should never have made it out into the wild.

I would have written the same reply if it was AMD or ARM though. In the end such problems must be detected in QA, no matter the company. It may be a sign of sloppy standards, or probably more likely that modern CPUs are too complex and to close to the "physical edge".

Well, that's very mature and noble of you on a personal level, but you are apparently very out-of-step with the AMD community and social media as a whole.

Remember - we are talking about 95% consensus that hitting advertised boost only when running NOPs in a loop, or running under LN2, meant that Zen2 wasn't "technically" defective or false advertising. You are an extreme minority here - this isn't an edge-case, this is how people actually think on social media.

And that's the problem - the public perception of all of this is shaped by the AMD social-media hype machine. Whether or not people think it's a "disaster" or a "nothingburger" is primarily determined by how much the AMD fans melt down about it. And they tend to blow up anything intel or NVIDIA to a massive degree.

Smeltdown was a disaster - the AMD prefetch+TLB vulnerability was a nothingburger, even though it was found by the same team and stated to be actually a worse exploit than Meltdown. AMD continued to even ship insecure-by-default and just slip in a small, non-explicit implication that if you cared you should enable KPTI manually (which of course hurts their benchmarks).

Over and over again.

you simply cannot broach almost any "is X thing actually important" without running into this issue - whether it's important depends on whether it favors AMD or not. People will literally flip stances topics overnight once it benefits AMD - more recently with framegen and latency. People will literally go from "unplayable fake frames" to "possibly the best part of FSR" literally overnight.

How do you have tech discussions in that atmosphere? On anything? I don't have a magic wand to fix that, but I'm just begging you here: this is going to be yet another Ayymd shitfest that nobody will remember in 2 years, just imagine that it's AMD in the hotseat and scale your reactions accordingly.

Just like New Worlds wasn't the end of NVIDIA (or even an NVIDIA-specific issue)... just like POSCAP/MLCC wasn't the end of 30-series... over and over again.

The chips have been out for literally 2 years already and the microarchitecture has been out for 3 years. It literally has to be a minor edge case or someone would have noticed it already - segfault on ryzen was noticed within months.

And this isn't a strawman or bait: you've already got people within this very thread winding up about "turning off turbo on intel" and shit like that. https://news.ycombinator.com/item?id=39480731

Was it proven that the issue still occurred with ECC RAM?

Yes, but then the processor will fail at another task during the game and corrupt some other memory. The only solution for unstable processor is to make it stable or replace.

Yes. Props to Oodle for not passing on the hot potato but trying to get the root cause fixed. This hack would have been the easy way out for them so their product doesn't get blamed.

quote: However, this problem does not only affect Oodle, and machines that suffer from this instability will also exhibit failures in standard benchmark and stress test programs.

It sounds like a hardware issue, i'm guessing over-agressive memory/cpu tuning, underpowered PSU triggering off behaviour etc. The fact that replacing the processor makes the problem go away does not in itself point to the processor as the issue - you may find that changing the memory also 'fixes' the problem.

Needs to be fixed by microcode, BIOS update or recall from Intel and partners

I'll add to the chorus of other responses that the whole computer is generally unreliable.

However, the really useful thing that this software could do is make the error message much better - explaining the likely thing that is causing the decompression to fail, and advising the computer should be fixed.

That error comes out of Unreal Engine proper, not Oodle, and is meant to be shown in cases when compressed shader data is corrupted in the game data files on disk in a way that hasn't been caught by package validation. Which is to say, it's not exactly common.

Oodle itself has some diagnostics hooked up to logs but none of that is user-facing. All the user-facing stuff needs to get handed off multiple times to get from low-level IO plumbing to somewhere that even knows how to display a user-facing error message to begin with.

The most common cause for that error message was, and continues to be (except on the relatively small number of affected machines), that compressed shader data on disk is corrupted. If and when we have a handle as to what actually causes the problem and a minimally-invasive fix (as opposed to the list of several different anecdotal "what if you try X?" that we got from Intel HW lab folks), we'll try to detect affected machines and point them to a website with instructions. For now, it's just a random error message that happens to frequently show up on machines encountering this issue, and all that would change if we changed the error message was that we'd confuse end users more and have to list two different error messages on that page instead of one.

We have the same check for silent data corruption in ClickHouse. After it detects a checksum mismatch, it also tries to check if it is caused by a single bit flip. If it is, we can provide a more precise diagnostic for a user about possible causes: https://github.com/ClickHouse/ClickHouse/issues?q=is%253Aiss...

Then the natural question arises: if we detect that it is a single bit flip, should we "un-flip" that bit, fix the data, and continue? The answer is: no. These types of errors should be explicit. They successfully help to detect broken RAM and broken network devices, that have to be replaced. However, the error is fixed automatically anyway by downloading the data from a replica.

(I'm the person who did most of the investigation.)

A relatively major realization during the investigation was that a different mystery bug that also seemed to be affecting many Unreal Engine games, namely a spurious "out of video memory" error reported by the graphics driver, seemed to be occurring not just on similar hardware, but in fact the exact same machines.

For a public example, if you google for "gamerevolution the finals crash on launch" and "gamerevolution the finals out of video memory", you'll find a pair of articles describing different errors, one resulting from an Oodle decompression error, and one from the graphics driver spuriously reporting out-of-memory errors, both posted on the same day with the same suggested fix (lower P-core max clock multiplier).

That's the problem right there in a nutshell. It's not just Oodle detecting spurious errors during its validation. Other code on the same machine is glitching too. And "just try repeating" is not a great fix because we can't trust the "should we repeat?" check any more on that machine than we can trust any of the other consistency checks that we already know are spuriously failing at a high rate.

Many known HW issues you can work around in software just fine, but frequent spurious CPU errors don't fall into that category.