Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they made a mistake without officially saying it?
I mean the whole world of Server or computer might not need ECC insanity was started entirely because of Google [1] [2] with news and articles published even in the early 00s [3]. And after that it has spread like wildfire and became a common accepted fact that even Google doesn't need ECC. Just like Apple were using custom ARM instruction to achieve their fast JS VM performance became a "fact". ( For the last time, no they didn't ). And proponents of ECC memory has been fighting this misinformation like mad for decades. To the point giving up and only rant about every now and then. [3]
[1] https://blog.codinghorror.com/building-a-computer-the-google...
Hello user, for an extra 10 dollars, would you like to guarantee that cosmic radiation never affects your computing experience, including having to reformat, reinstall, or otherwise reboot your phone/computer randomly when one day something just doesn't work anymore?
Was it an update? Was it cosmic radiation? Was it a bad capacitor? Who knows!
Why do you think so many problems are solved by rebooting? Sure 99 out of 100 might be software bugs, but the other 1 out of 100 is straight up cosmic radiation, and that 1 percent is growing more and more every year as software becomes more robust and bug free with better tooling.
So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see, only addressing a single one of the root causes that trigger that same result, and in the end you're not even completely fixing it, just reducing the already infinitesimal odds it happens.
It will become mainstream when manufacturers just decides to include it everywhere and not really give the user a choice. Apple is a prime candidate for a company with enough clout to afford this.
>2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year".
https://stackoverflow.com/questions/2580933/cosmic-rays-what...
I believe bitflips are far more common than people realize. I see weird shit all the time from my users that is only explainable due to bizarre software bugs or bitflips.
Don't get me wrong, I'm all for ECC memory, I practically showed the value of ECC after intentionally running the same multi-day Matlab job several times on a non-ECC machine with different results just to prove this very point. But you use your deeper knowledge to assume what someone with shallow knowledge and interest wants or needs, and that will almost always be off mark. Your car does not have a roll cage. Normal computers do not have ECC memory.
For ECC, there's no reason to really expect that adding ECC to everything would really make things too expensive or slow long term. For a long time the reason why desktops did not have ECC was pretty much because intel wanted people who really need ECC to buy Xeons.
Exactly like how we try to recreate the effect of ECC hardware but without its "weight" (cost, complexity). For a problem that's not nearly as visible or life and death.
> too expensive or slow long term
Not "too" expensive long term still means more expensive especially short term, like when the person buys it. For an issue no average consumer is actually frustrated about. Why are we still debating why those consumers don't care about ECC? How is it different from hot swappable RAM for the average Joe? They'll upgrade or expand their RAM more often than they'll get frustrated about the effect of bitflips. So why not have hot swappable memory? Because as much as I'd love hot swappable anything, regular consumers neither care about it nor want to pay for it.
I don't know the same argument could be made and ecc be swapped with surge protectors and people still purchase those.
I was buying DDR4 RAM last week and the cost for a 4x8GB ECC+Registered DDR4 3200 RAM kit for a Xeon W build I’ll be doing soon was about 5% more expensive than for a kit of otherwise identical RAM kit for a non-ECC Core i7 rig - which also came with tacky RGB LED heatspreader - about $230 in total for each of them.
The fact the pricing was so similar does make me wonder about the claims that non-ECC RAM is really ECC RAM just without the CPU/MMU being made aware of it - I think that’s possible if the RAM is already Buffered/Registered.
Who claims that? And can't you just count the chips?
Intel's decision could just as well have been made by a socialist government's regulatory body (in the form of "minimum requirements for consumer chip" legislation), or by a committee of decision makers in an employee-owned company. We may never know, because we only have 2-4 data points and they're all basically capitalist.
Honestly, I'm not sure it was intended to be a serious comment. It could just as easily have been "...failure of human cognition and Obama."
https://semiengineering.com/what-designers-need-to-know-abou...
There has been a lot of debate regarding this that was summarised in this post -
On-die ECC is going to be a standard feature for DDR5. I'm not aware of any indication that anyone has implemented on-die ECC for DDR4 DRAM, and Hynix at least has made clear statements that on-die ECC is new for their DDR5 and was not present in their DDR4.
On-die ECC DRAM is already here with us.
There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?
When America Online was buying EV6 servers as fast as DEC could produce them, they used to see about about 1 double bit error per day across their server farm that would reboot the whole machine.
DRAM has only gotten worse--not better.
Bit flips (for all reasons) occur in buses, registers, caches, etc. Anything that has state can have state changed incorrectly.
This is why filesystems like ZFS exist and storage formats have pervasive checksums.
https://www.newyorker.com/magazine/2018/12/10/the-friendship...
Edit: found one with a quick search. https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...
And https://www.researchgate.net/publication/262273269_Bitsquatt...
But I'm not entirely convinced that most of these requests weren't just typos. Though those requests with mismatched Host header surely were true bitflips.
And I am beyond certain they were not typos; the requests were for long URLs that no one would be typing by hand. A few were for services that humans never enter urls into (like the Windows crash reporter).
I can't find it right now, but I also saw someone doing this with some version of Google Now (or what it was back then) and serving different assets into it.
Random bit flips happen on client machines and on routers.
If there are enough requests for a domain name, some of those requests will be subject to one one of those bit-flips.
“I’ve heard of defensive programming, but never adversarial memory.” — Ben Gomes
> I've never thought of defensive programming in terms of adversarial memory.
Generally this goes against software engineering principles. You don't try to eliminate the chances of failure and hope for the best. You need to create these failures constantly (within reasonable bounds) and make sure your software is able to handle them. Using ECC ram is the opposite. You just make it so unlikely to happen, that you will generally not encounter these errors at scale anymore, but nontheless they can still happen and now you will be completely unprepared to deal with them, since you chose to ignore this class of errors and move it under the rug.
Another intersting side effect of quorum is that it also makes certain attacks more difficult to pull off, since now you have to make sure that a quorum of machines gives the same "wrong" answer for an attack to work.
Using consumer hardware and making up reliability with redundancy and software was not a bad idea for early Google but it did end up with an unforeseen cost. Just a thousand machines in a cosmic ray proof bunker will end up with memory errors ECC will correct for free. It's just reducing the surface area of "potential problems".
That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost.
AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level...
The original IBM PC AT had parity in memory.
I don't know if AMD really intended to break Intel's pricing model. Their higher end Ryzen chips you'd use in servers and capital W Workstations don't seem to have a huge price difference from equivalent Xeons. Even if they're a bit cheaper you still need a motherboard that supports ECC so it seems at first glance to be a wash as far as price.
That being said if I was putting together a machine today it would be Ryzen-based with ECC.
You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...
[0] https://www.asrock.com/mb/amd/a320m-hdv%20r4.0/index.asp#Spe...
This is the legacy of Intel's policies.
As such features that are unattractive to the regular consumer go into workstation/enterprise offerings where the buyer understands what they're buying and why.
Citation needed.
I would bet that your typical ram-purchasing consumer is not seeing or even considering the existence of the ECC model.
> ECC realistically adds to the BOM (board, modules) more than LEDs do. So the price goes up with seemingly no benefit for the user.
LEDs are a great opportunity to increase profit margin, so I'm not sure about your price conclusions.
> Citation needed.
It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price.
> LEDs are a great opportunity to increase profit margin, so I'm not sure about your price conclusions
You confused manufacturing costs, price of the product, and profit margins. LEDs cost far less to integrate than ECC but command a higher price premium (thus better profit margins) from the regular consumer. Again supporting my statement that even if presented with 2 absolutely identical parts save for ECC vs. LEDs the vast majority of consumers will go for LEDs because they don't care or know about ECC.
You're making a claim about what people would choose. If you have no related data, and logic could support multiple outcomes, then a claim like that is basically useless.
> You confused manufacturing costs, price of the product, and profit margins.
I'm not sure why you think this.
> Again supporting my statement that even if presented with 2 absolutely identical parts save for ECC vs. LEDs the vast majority of consumers will go for LEDs because they don't care or know about ECC.
Sure, if you don't tell them that it's ECC they won't pick the ECC part.
If you actually do a fair test, and put them side by side while explaining that one protects them from memory errors and the other looks cooler, you can't assume they'll all pick the LED.
When people never even think of ECC, that is not evidence that they wouldn't care or know about it in a head-to-head competition.
My claims are common sense and supported by the real life: regular people don't know what ECC is, and those who do find the problem's impact is too minor to get palpable benefits from fixing it. Why are you being pedantic if you aren't actually going to bring arguments at the same level you expect from me?
> If you actually do a fair test, and put them side by side while explaining that one protects them from memory errors and the other looks cooler, you can't assume they'll all pick the LED.
Isn't this exactly the kind of claim you yourself characterize one paragraph above as "useless" because "you have no related data, and logic could support multiple outcomes"? Sure, if people were more tech-educated then my assumption might be wrong. But people aren't more educated so...
The benefits of LEDs are hard to miss (light) all the time. The benefits of ECC are hard to observe even in that fraction of a percent of the time. Human cellular "bitflips" happen every hour but they don't visibly affect you so you also consider it's not an issue that demands more attention, like constant solar protection. People aren't keen on paying to solve problems they never suffered from, or even noticed, especially when you tell them they happen so often still with no obvious impact. Unless they have no choice, like OEMs selling ECC RAM only devices.
Sell me ECC memory when my (actual real life) 10 year old desktop or 5 year old phone never glitched. Sell me ECC RAM when my Matlab calculations come back different every time. See the difference?
> When people never even think of ECC, that is not evidence that they wouldn't care or know about it in a head-to-head competition.
Well then, I guess none of us has any evidence except today people buy LEDs not ECC RAM. Educate people or wait until manufacturing process and design are so susceptible to bitflips that people notice and it will be a different conversation.
Regular people aren't given the choice! The things you're quoting about the real world to support your argument are incompatible with a scenario where someone is actually looking at ECC and LED next to each other. And I'm not being "pedantic" to say that, it's a really core point.
> Isn't this exactly the kind of claim you yourself characterize one paragraph above as "useless" because "you have no related data, and logic could support multiple outcomes"?
A claim of a specific outcome is useless. "you can't assume" is another way of phrasing the lack of knowledge of specific outcomes.
> Sure, if people were more tech-educated then my assumption might be wrong. But people aren't more educated so...
It's the kind of thing that can go on a product page. But first someone has to actually make a consumer-focused sales page for ECC memory, and the ECC has to be plug-and-play without strong compatibility worries.
And just like when LEDs spread over everything, it's something that you can teach people about and create demand for with a bit of advertising.
> Sell me ECC memory when my (actual real life) 10 year old desktop or 5 year old phone never glitched. Sell me ECC RAM when my Matlab calculations come back different every time. See the difference?
That's a clear picture of one person. But "never glitched" is a very dubious claim, and you can't blindly extrapolate that to how everyone would act.
It would definitely be great to have more reliable hardware generally available and at less of a price premium.
1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload
2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or why. Their SREs are world-class and may be able to deduce if this was a fluke (orders of magnitude less likely than a single bit flip), if a workload caused it, or if hardware caused it.
The advantage the 3 major cloud providers have is scale. If a Fortune 500 were running their own datacenters, how likely would it be that they have the same level of visibility into their workloads, the quality of SREs to diagnose, and the sheer statistical power of scale?
I sincerely hope Google is not simply silencing bitflip corrections and detections. That would be a profound waste.
Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not sure if it is even possible.
The state was quite helpful, yes–for x86 it seems like a "clean slate" shellcode would be quite difficult, if impossible, to achieve as we saw. However, I am left wondering how other ISAs would fare…perhaps worse, since x86 is notoriously dense. But maybe not? The fixed-width ones would probably be easy to try out, at least.
[1] https://developer.arm.com/docs/ddi0596/h/top-level-encodings...
Was any code dumped anywhere?
I found this which corroborates everything you're saying but provides no further details: https://www.cspensky.info/slides/defcon_27_shortman.pdf
Coverage of the finals is usually much less detailed, unfortunately, since the number of teams is much smaller and the challenges don't necessarily go up. However, https://oooverflow.io/dc-ctf-2020-quals/ has a couple more writeups linked from it; https://dttw.tech/posts/SJ40_7MNS#proof-by-exhaustion from PPP and http://www.secmem.org/blog/2019/08/19/Shellcoding-and-Bitfli... from SeoulPlusBadass.
Binary bitflip resilience is really cool. The radiation-hardened-quine idea (https://codegolf.stackexchange.com/questions/57257/radiation..., https://github.com/mame/radiation-hardened-quine) is cool, but these source-based approaches depend on a perfectly functioning and rather large (Ruby, V8, whole browser) binary stack. A bitflip-protected hex monitor or kernel, on the other hand...
There is an OS that pretty much fits the bill here. There was a show where Andrew Tanenbaum had a laptop running Minix 3 hooked up to a button that injected random changes into module code while it was running to demonstrate it's resilience to random bugs. Quite fitting that this discussion was initiated by Linus!
Although it was intended to protect against bad software I don't see why it wouldn't also go a long way in protecting the OS against bitflips. Minix 3 uses a microkernel with a "reincarnation server" which means it can automatically reload any misbehaving code not part of the core kernel on the fly (which for Minix is almost everything). This even includes disk drivers. In the case of misbehaving code there is some kind of triple redundancy mechanism much like the "quorum" you suggest, but that is where my crude understanding ends. AFAIR Userland software could in theory also benefit provided it was written in such a way to be able to continue gracefully on reloading.
Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.
One advantage of microkernels is that the "watcher" is so small that it could be run directly from ROM, instead of loaded into RAM. QNX has advocated that route for robotics and such in the past.
Minix may not be the best example of the type. While it is a microkernel, it's real world reliability has been poor in the past. More mature microkernel operating systems like QNX and OpenVMS are better examples.
Nitpick/clarification: it currently supervises the security posture, attestation state and overall health of several billion(?) Intel CPUs as the kernel used by the latest version of the Management Engine.
If ME is shut down completely apparently the CPU switches off within 20 minutes. Presumably this applies across the full uptime of the processor, and not just immediately after boot, and iff this is the case... percentage of Intel CPUs that randomly switch off === instability/unreliability of Minix in a tightly controlled industrial setting.
I used to occasionally boot into QNX on my desktop in college. It was a very responsive and stable system.
Hypervisors are, to a first approximation, microkernels with a hardware-like interface. All of this kernel bypass work being done by RDBMSes, ScyllaDB, HFTs, etc. is, to a first approximation, making a monolithic kernel act a bit like a microkernel.
Why something hasn't been done is always a hard question to answer, since to succeed a lot of things have to go right, and by default none of them do. But one thing is that microkernels were more trendy in the 90s - r&d people are mostly doing things like "the cluster is the computer", unikernel, exokernel, rump kernel, embedded (eg tock), remote attestation since then (I'm not up to date on the latest).
You might be referring to the previous versions. Minix 3 is basically a different OS, it's more than an educational tool - in fact it's probably running inside your computer right now if you have an Intel CPU (it runs Intel's ME chip - for better or worse).
As far as bitflips are concerned, having the critical kernel code occupy fewer bits reduces the probability of a bitflip causing an irrecoverable error.
(I'll archaic brag a bit by mentioning I used to be a heavy user of Minix - my floppy images came in over an X25 network - and saw Andy Tanenbaum give his Minix 3 keynote at FOSDEM about a decade ago. I'm a big fan.)
Anyway, while reducing risk this way is laudable, and will improve your fleet's health, as per TFA it's a poor substitute, with bad economics and worse politics behind it, than simply stumping up for ECC.
I'll also note that, for example, Google's sitting on ~3 million servers so that ~4k LoC just blew out to 12,000,000,000 LoC -- and that's for the hypervisors only.
Multiply that out by ~50 to include VM's microkernels, and the amount of memory you've now got that is highly susceptible to undetected bit-flips is well into the mind-blowing range.
Doing quorum at the computer level would require synchronizing parallel computers, and unless that synchronization were to happen for each low level instruction, then it would have to be written into the software to take a vote at critical points. This is going to be greatly detrimental both to throughput and software complexity.
I guess you could implement the quorum at the CPU level... e.g. have redundant cores each with their own memory. But unless there was a need to protect against CPU cores themselves being unreliable, I don't see this making sense either.
At the end of the day, at some level, it will always come down to probabilities. "Software engineering principles" will never eliminate that.
There are a lot of seemingly high-level problems that are solved (ingeniously) in hardware with very simple, very low-level solutions.
My first employer out of Uni had an option for their primary product to use a NonStop for storage -- I think HP funded development, and I'm not sure we ever sold any licenses for it.
Generally, it would make the most sense to kill the process if the corrupted page is data, but if it's code, then maybe re-load that page from the executable file on non-volatile storage. (You might also be able to rescue some data pages from swap space this way.)
What would be interesting is if userspace could mark a region of memory as recomputable. If the kernel is notified of memory corruption there, it triggers a handler in the userspace process to rebuild the data. Granted, given the current state of hardware; I can't imagine that is anywhere near worth the effort to implement.
I believe there's already some support for things like this, but intended as a mechanism to gracefully handle memory pressure rather than corruption. Apple has a Purgeable Memory mechanism, but handled through higher-level interfaces rather than something like madvise().
The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.
You do the comparison on multiple nodes too. Get the calculations. Pass them to multiple nodes, validate again and if it all matches, you use it.
Recursion, see recursion.
ECC memory is just as fast as non-ECC memory, and only cost a little more.
I am totally for ECC and was flabbergasted when it went away. But the article makes sense since I remember Intel pushing hard to keep it out of the consumer space. The freaking QX6800 didn't support ECC and it retailed for over a grand.
Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the software being stored in that memory being robust against cosmic rays. And when you have thousands of machines with petabytes of RAM, those bitflips do happen. Google wasted many man-years tracking down corrupted GFS files and index shards before they finally bit the bullet and just paid for ECC.