The problem is right now DDR4 devices that work correctly in that regard, do not exist. Likely the best mitigation for any sensitive application, even so lightly, is DDR4 with ECC (even if it may not be enough, it is vastly better than nothing, and not just because of rowhammer)

And I have no idea if the internal "ECC" of standard DDR5 helps or not. It is not intended for regular ECC level of reliability anyway. (And I have seen discussion about likely bitflips detected in crash dumps of M1 Pro devices)

So, as much as I would like to return defective devices, I would probably be left with no computer, no smartphone, etc.

According to older paper [0] ECC can also be bypassed after reverse-engineering the mitigation in DDR3 DIMMs. Also:

> “DDR4 systems with ECC will likely be more exploitable, after reverse-engineering the ECC functions,” researchers Razavi and Jattke said

> What if I have ECC-capable DIMMs? Previous work [1] showed that due to the large number of bit flips in current DDR4 devices, ECC cannot provide complete protection against Rowhammer but makes exploitation harder.

[0] https://cs.vu.nl/~lcr220/ecc/ecc-rh-paper-eccploit-press-pre...

[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8835222

While it can not protect from bit flips it can most likely warn about attack in progress due to a large number of bit flips.

> The problem is right now DDR4 devices that work correctly in that regard, do not exist.

Yes, and this is actually the problem. Without safe hardware, it is almost impossible to write safe software.

Meanwhile you can still buy DDR3 --- used --- from various shops in China, and it'll be 100% perfect and Rowhammer-free because it was made before the insane process shrinkages that caused it.

(Have done this a few times. Was hesitant initially because of the low price and used nature, but when the first stick passed MemTest86+ including RH perfectly, I was convinced. Half a dozen more sticks later and still good. But maybe they've started to run out of the good old stuff now...)

> you can still buy DDR3 --- used --- from various shops in China, and it'll be 100% perfect and Rowhammer-free because it was made before the insane process shrinkages that caused it

That's demonstrably false. I remember rowhammer being thought of and demonstrated on DDR3 modules.

[0] https://hammertux.github.io/rowhammer-ffs-ddr3

Rowhammer started appearing in newer DDR3; see the original paper (figure 3): https://news.ycombinator.com/item?id=8713411

The tl;dr is that everything 2009 and older is good, 2010-2011 is possibly bad, 2012 and newer is almost certainly bad.

> Yes, and this is actually the problem. Without safe hardware, it is almost impossible to write safe software.

Software that attempts a hostile rowhammer attack aims to be unsafe by definition. It's not a matter of trying and failing to be safe. Rather, we're seeing that it's very difficult to build hardware and software that prevents hostile code from running amuck: if the attacker can run hostile code on your computer at all, you have already lost. That's bad news for cloud hosts and confirms what we already knew about web scripting, but doesn't seem all THAT bad for the rest of us.

Of course it's possible to build computers with no DRAM at all (i.e. only SRAM, which costs more). That is not economically sane for big servers, but might be the right thing to do for security sensitive components.

The sizes of current CPU caches is enough to run a lot of code…

Maybe a paranoid app could try to allocate the memory adjacent to any sensitive bits as a buffer? That would be pretty difficult to do but might be possible if you are tremendously paranoid and have a good way to examine the hardware.

You'd have to have the operating system's cooperation, because it may be mapping individual 4k pages all over the place, and it's the only thing that has a chance of knowing how the memory is laid out on the actual chips.

And the CPU and the chipset's cooperation. In many cases it is even a trade secret how data is striped across the different slots/banks.

<i> it is even a trade secret how data is striped across the different slots/banks </i>

If that is true, vs just the usual "you don't need to know" trash common these days, its crazy. A pretty good picture can be built up with just software timing analysis, but its hardly rocket science to hook a logic analyzer/scope up and determine bit swizzling and page/bank interleave. Plus, many of the more open BIOS vendors have tended to provide page/row/controller/socket interleave options for quite a while, partially because it can mean a 5-10% uplift for some applications to have it set one way or the other. Its been one of those how do you tune your memory system options for a couple decades now.

Doesn't the same thing to exploiting rowhammer though? If you're after specific data, how would you know what physical address to use, even if you knew the physical address of the target data?

Yes, but there is a difference between an attack being run by a low level hardware hacker vs. the software being run by average users. This is why I said it would be hard, you would need to encode a lot of low level hardware knowledge into your application.

This might only be useful in very specialized circumstances, like with kernel support on only a handful of carefully chosen hardware platforms. However, someone like Apple could do this on their systems, as they control both the kernel and the hardware. Sensitive memory locations could be cordoned off in special zones with buffering to prevent Rowhammer type attacks.

Or use different banks of DRAM for different security contexts, managed via NUMA?

I wonder how easy that would be to reason about though. I really don't know much about modern DRAM circuitry I feel like it might be abstracted away to some level, also there are bios-tweaky settings that might make a difference (i.e. things like channel configuration and/or bank interleaving, if that's still a thing.)

IDK though. Maybe there's a way. I've worked on code that uses padding around a data structure to ensure it has it's own cache line. Maybe if you allocate a large enough contiguous block you'll be OK?

If your app is particularly paranoid you could just double your memory usage, and verify errors against the clone.

Yep, though I agree with the parent, if a RAM returns data from a location that is different from what is stored in that location, assuming all recommended timings have been followed, then that RAM is defective. If that means they should be recommending a refresh after every single operation, then that's what it means.

In other words, the whole industry sits on a bed of lies at this point and it's only because government is technically incompetent that we haven't seen the world's biggest class-action.

> In other words, the whole industry sits on a bed of lies at this point and it's only because government is technically incompetent that we haven't seen the world's biggest class-action.

That's a pretty bold claim - if it's accurate, can you point at the government regulation that precludes a class action lawsuit? I would assume it's something in the T&C or EULA for the hardware rather than a regulation?

Bold but sadly correct. The industry runs on misdirection and deception.

If users knew better, we'd have the equivalent of another Pentium FDIV bug.

The whole industry built this bed of lies by their own hands, I don’t see how it’s government’s fault.

I think this illustrates the cloud is unacceptable for anything more than storage and retrieval.

All computed results from data science must include steps and code to verify locally.

It calls into question privacy on federated networks and crypto networks; any node can be manipulated locally to change payload outputs on delivery, reveal secrets, disrupt workloads.

This makes sense to a lot of folks in computer engineering and physics, versus abstract software. No physical theory I know of offers any guarantee our arbitrary computing machines will ever be securable. We put fart pipes on Hondas.

Science proves it’s titillating smoke and mirrors once again. Still waiting for nuclear rocket cars.

I think this proves further as well why general computing chips need to be replaced with workload specific designs, where the anticipated inputs are well known and no vague logic paths to intentionally allow software monkey patching ever ship.

Security doesn’t scale at a price point that private sector companies could typically afford.

Perhaps we fail at pricing security into the value of a company, or maybe that’s what risk appetite is about.

The problem is that you can get away with minimal security for a long time. Sure, if you get hit, shit hits the fan. But by then, it's quite likely that all competitors that spend money and time on security and good infrastructure are long gone.

This is worsened by the fact that it's very hard for laypeople to assess the security of a specific application and that, by now, "cyberattack" has become common enough that it's easily accepted as an excuse.

Which is why certifications, audits, and minimum mandated standards are critically important.

The market just yawns at this stuff, until it gets fragged. Then it forgets and the cycle repeats.

> Which is why certifications, audits, and minimum mandated standards are critically important.

Not sure about that. All the security standards want me to run software written in an unsafe language as root on every device, intentionally parsing malicious inputs continuously.

That’s not making anything safer.

Pretty clearly, the standards have to be effective and well-designed. And yes, there are problems with that.

But the point remains that markets do very poorly at rare and/or cumulative risks. And that's the comparison I'm making. The market of and by itself will give you a race to the bottom in standards.

A longer-term view, whether through government regulation and oversight, social suasion, religious morality and ethics, or (possibly) insurance-oriented risk management (yes, a market mechanism, though something of an exception to the rule), will typically operate by the mechanisms I've described above. That there may be poor implementations doesn't obviate the fact that there can also be good ones, and that that's the goal we're aiming for.

Probably, at least some part of a modern financial sector including startups has many things in common to pyramid schemes.

> I think this illustrates the cloud is unacceptable for anything more than storage and retrieval.

You can run in dedicated tenancy where you have the whole machine or a metal configuration where you also have the whole machine.

That would ruin providers' user:hardware ratios, one of the foundational principals of cloud computing.

On the contrary, certain CSPs did this already (quietly) and further, had already developed hardware mitigations for things like meltdown, spectre and rowhammer.

But did they manage to keep their prices low?

(And those who employed hardware mitigations wouldn't have a problem with user:hardware ratios, would they?)

> I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this.

Is this an individual stick bug, or a design bug akin to Spectre? Given that they tested 40 different sticks and seemed to find effective patterns for all of them make me think it's the latter. In which case, just return all your computers?

Yeah, this is a problem with high density DRAM. Fixes are possible but difficult and expensive. In the cutthroat DRAM market nobody seems willing to put in the engineering effort to develop a solution and then manufacture it. It would have to sell at a premium, in a market where it's difficult to even find ECC memory at times.

The article states that ECC isn't sufficient to solve Rowhammer, but it does make the attack harder. If you are looking for a hardware solution it is your only option currently, even if it is imperfect.

I imagine you are right that consumers don't care or know enough to put pressure on manufacturers, but what about big companies like the FANGs? Or Government agencies like NSA etc? What are they doing about it? I imagine intelligence agencies are well aware of the issue and have or are trying to get solutions to protect their own infra.

Off topic but shouldn’t FAANG really be referred to as MAAAN now?

Google -> Alphabet

Facebook -> Meta

Meta, Apple, Amazon, Alphabet, Netflix. MAAAN.

Google still exists. Facebook still exists. Meta/Alphabet are mainly holding companies. For example, I received reach outs from recruiters for Google and Facebook today. Not from Meta and Alphabet. Maybe once Meta and Alphabet deliver on things and build brand recognition, it should change. Until then, I'd vote "no".

Now I can add "I have never worked for the MAAAN" with pride on my business cards!

Every now and again I worry I've grown up. Then I realize there is no grown up, just a Matroyshka doll of more and more eclectic subtexts

MANGA

M - Meta A - Apple N - Netflix G - Google A - Amazon

I'm now adopting this, thanks

AMANAM

Apple Microsoft Amazon Netflix Alphabet Meta

If you're doing that: M3AN.

I like MAGMA more

Right, the idea was to swap Netflix for Microsoft, giving MAGMA or GAMMA:

https://news.ycombinator.com/item?id=29042753

If you detect flips you can send it back, it's violating the advertised interface.

Sure, but if it's possible to cause bit flips in basically all currently available components, then what's the point? Return all of your RAM and have a non-functional computer?

Can't get bit flips on my RAM if I have no RAM.

So you just.. Send back all of your RAM all the time because they are all susceptible to bitflips?

This is not a tenable solution.

It happened because at some point in the past (around 10 years ago I believe), the memory vendors decided that it was ok to design product with known "pattern sensitivity", because...I suppose they made more money that way and no customer complained loudly. Pattern sensitivity in memory chips is a very old problem, and previously was treated as a fatal design error, fixed and the affected chips used for parking lot infill. But today every chip sold is affected. Basically the same as Boeing and the MAX, just fewer people killed.

> Pattern sensitivity in memory chips is a very old problem, and previously was treated as a fatal design error

You seem to know more about this than I do. Where might one go to read more about this? Are there whitepapers, design docs, other resources which could be used to assert positively that the industry used to consider this a defect?

The problem is that apparently the authors of memory testing programs were somehow convinced to hide the severity of this issue when it first appeared, claiming things like it being a rare edge case and that too much RAM would fail.

I believe newer versions of MemTest86+ have Rowhammer tests but it's disabled by default.

See my previous comment on this matter:

https://news.ycombinator.com/item?id=12410274

MemTest86 already has a rowhammer test, not sure how it compares to blacksmith but some sticks I have do fail that test and can only be mitigated by setting tREFI extremely low (while also taking a large performance hit).

Most of the higher end hynix/micron/samsung sticks I've tried do not fail at JEDEC or XMP after 7+ passes.

Pretty sure the passmark version does https://www.memtest86.com/compare.html

Except that the distro's are still shipping an older version due to license issues (or something like that AFAIK).

You'll be sending back all of your RAM then.

Google is working on platforms to make this easier to explore this problem, see https://opensource.googleblog.com/2021/11/Open%20source%20DD...

> I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this.

How do you run a memory tester after you have returned all CPUs after Meltdown and all the subsequent microarchicture bugs? I'd call all modern CPUs faulty products. Some more than others.

If it's a problem with chips becoming higher density, I wonder if older hardware will become popular in circles that care more about security than performance per rack unit.