News for hackers. Stuff that matters.
Re:Google says Rowhammer attacks are gaining range as RAM is... by marcan_42 on Friday May 28, 2021 @10:58PM (#27322700)
The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes.

The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND Flash, HDDs, optical media, etc. We've been doing this for decades. Even floppies at least had error detection (checksums). All high speed transmission protocols also do error detection and re-transmission, except HDMI (because I guess nobody cares about the odd corrupted pixel). Same for anything going over radio.

Once you have ECC, your error rate margin goes up by many orders of magnitude. Worst case these attacks turn into a DoS as ECC fails to correct a badly corrupted word, and most of the time it will just work. Plus you can detect attack attempts as the memory controller will report increased error rates.

It's not possible to design modern RAM that is perfectly reliable. That's just physics. We know how to solve this problem. We just aren't doing it because Intel thinks consumers don't deserve reliable RAM. The only other option is doing ECC on-chip in the RAM itself, and that has other efficiency trade-offs which probably make it not worth it, at least not with DDR style interfaces (it might've made sense in an FB-DIMM style world where the memory controller is in the RAM)

Re:Google says Rowhammer attacks are gaining range as RAM is... by Laforet on Saturday May 29, 2021 @12:18AM (#27322994)
Not here to defend Intel (their decision to remove ECC support from X299 chipset pretty much tanked their own HEDT platform), but I often wonder how much of the current disregard of ECC is the fault of the system integrators.

As of now they would gladly ship configurations with single channel memory and under-par cooling to save a few bucks, while these cut corners could hurt performance by as much as 30% due to insufficient memory bandwidth and thermal throttling.

Even if ECC is available everywhere I doubt any OEM will use it without charging their users a premium. This is kind of what has been going on with AMD MSDT platforms: ECC support is in the CPU but many motherboard does not even implement the necessary memory traces to enable it, and software support is even messier to the point that most people can't be bothered with it.

Re:Google says Rowhammer attacks are gaining range as RAM is... by marcan_42 on Saturday May 29, 2021 @01:17AM (#27323211)
This is a self-fulfilling issue; since Intel does not support ECC, ECC is not mainstream, so mainstream platforms don't care about ECC. It would cost motherboard manufacturers ~nothing to support this properly, so if it were accepted as a thing they would, and it would eventually be a requirement for newer RAM technologies. ECC should've been mandatory since about the DDR2 era, practically speaking. That it isn't is a collective failure of the computer industry, and Intel takes a big part of the blame.

At least I had no issues getting ECC to work on my Threadripper box, but sure, that's HEDT.

Re:Google says Rowhammer attacks are gaining range as RAM is... by staticassertion on Saturday May 29, 2021 @12:32AM (#27323044)
Maybe while both ECC and non-ECC are both being sold, but that would be a temporary situation. The goal would be to have non-ECC memory be effectively deprecated, so the next revision of the spec enforces ECC as a requirement.
Re:Google says Rowhammer attacks are gaining range as RAM is... by formerly_proven on Saturday May 29, 2021 @12:36PM (#27326361)
Doing ECC on-die (as in DDR5) does solve issues related to the storage itself, and the access timing of the array, but still leaves data vulnerable to corruption on the memory bus itself. Memory is indeed the only place were we are moving and storing data and it is commonplace to use zero layers of data protection; everywhere else we have at least one, often multiple, checksums and storage always has had forward error correction for a very long time, regardless of medium. But of course, generally speaking, all data processed by a computer moves through the main memory.
Re:Google says Rowhammer attacks are gaining range as RAM is... by Zenst on Saturday May 29, 2021 @07:55AM (#27324672)
> The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes.

Whilst Intel did play a part, when they offered it as standard the uptake in the consumer market was low and the price factor and need at most consumer level was not there.

Today it is there, though the consumer mindset is unaware of it and I do foresee a time in which either some in-the-wild exploit that makes media traction and they roll with it event. Then there will be a sudden panic rush with media rhetoric akin to the run-up to Y2K and drama headlines how everybody needs to buy a new computer. Now if that happened - Intel on the desktop would sure take a wack as it is currently.

Then the whole supply and demand aspect that factors into the cost and alas - until there is some mass consumer awareness.

Well, until then for the majority consumers, computer memory is just computer memory and for most of those, they don't even look into things like memory speed or dual-channel memory even. Which see's your DELL's et all selling systems with one large stick of RAM even today as your average consumer just don't know any better. So the move to ECC for consumers will need en event or large player marketing it. Maybe a balance of the two.

Just the aspect of size of memory today in computers, let alone density and speeds, bit errors become far more likely and that is without the security aspects at play. So for stability alone ECC is becoming more and more justifiable.

Re:Google says Rowhammer attacks are gaining range as RAM is... by rasz on Sunday May 30, 2021 @02:07AM (#27330985)
Please, The ONLY time Intel offered ECC as standard on its products was during brief 486 period when all chipsets the offered were considered highend and their chipset market share was in single digits.

ECC was standard on first IBM PC 5150, on PS/2 line, on pretty much all 286 clones etc.

Intel market segmentation shenanigans started around 1995 with Pentium line, with ECC absent from all consumer oriented product lines.

Re:Google says Rowhammer attacks are gaining range as RAM is... by Zenst on Tuesday June 01, 2021 @11:09AM (#27355547)
True, the whole dip into RAMBUS kinda saw them shift things and when they came back to less niched out RAM, they split things business and consumer.

however, been some odd glimmer of hope for the cheaper end for ECC suport from Intel - https://www.truenas.com/community/threads/surprise-surprise-...

What really needs to happen is all these AMD Ryzen users to buy into ECC, help get the price down and more so, push Intel in another tickbox catchup in that market.

Re:Google says Rowhammer attacks are gaining range as RAM is... by rasz on Tuesday June 01, 2021 @01:36PM (#27358294)
Intel cut ECC from its consumer chipsets almost 10 years before Rambus.
Re:Google says Rowhammer attacks are gaining range as RAM is... by femiagbabiaka on Saturday May 29, 2021 @11:07AM (#27325720)
I thought ECC wasn’t effective protection against Rowhammer? Agreed that it is incredibly useful otherwise. https://www.vusec.net/projects/eccploit/
Re:Google says Rowhammer attacks are gaining range as RAM is... by marcan_42 on Saturday May 29, 2021 @10:37PM (#27330161)
ECC helps a lot, and combined with memory encryption (which is another thing we should be doing) it effectively mitigates the issue, as then you can't target specific bits any more, so your chances of getting past ECC without causing hard errors are negligible.
Re:Google says Rowhammer attacks are gaining range as RAM is... by userbinator on Friday May 28, 2021 @11:55PM (#27322913)
ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation.

Plus you can detect attack attempts as the memory controller will report increased error rates.

Now you've turned specific access patterns into "attack attempts", and by discriminating against them as such, entirely destroyed a fundamental part of general-purpose computing. What's next, CPUs that are only guaranteed to do certain operations correctly, and everything else is considered an "attack"? That is NOT a direction that we should be heading. Hell fucking no!!!

RAM should always hold what was last written to it, under all conditions of software accesses. To expect or imply that anything less is acceptable, is to destroy one of the foundations on which general-purpose computing is built.

NAND Flash

Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance? That's a rant for another thread...

We know how to solve this problem.

This problem didn't exist 15 years ago. It only got created because the RAM companies have forsaken correctness, and somehow managed to convince everyone else that it's not their own fault.

Re:Google says Rowhammer attacks are gaining range as RAM is... by staticassertion on Saturday May 29, 2021 @12:38AM (#27323064)
> Now you've turned specific access patterns into "attack attempts",

We already have error correction detection available to us and it's in use today.

> RAM should always hold what was last written to it, under all conditions of software accesses.

OK? That's what anyone who is advocating for ECC is already advocating for. ECC is error correcting up to 1 bit and error detecting up to 2 bits. The goal is to ensure that the property you've described holds.

> It only got created because the RAM companies have forsaken correctness,

This problem exists because... physics. Do you actually have a design for RAM that can match today's cell density while somehow providing perfect isolation? Saying this problem didn't exist 15 years ago is like saying that malware didn't impact the abacus - RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency.

And for reference, hamming codes (which back ECC) were created in the 1950s for error correction in punch cards. This is not a new problem.

ECC is the existing, obvious solution to this problem, it just wasn't built for this threat - but it's an excellent, proven method for correcting bit flips. Even without being designed to prevent rowhammer it already makes the attack much slower, less likely to succeed, and easier to detect (just monitor for a massive spike in ecc corrections).

Trying to design high density RAM that maintains perfect isolation seems pointless. Improving ECC or ECC-like approaches is likely going to be much more effective.

Re:Google says Rowhammer attacks are gaining range as RAM is... by beebeepka on Saturday May 29, 2021 @04:55AM (#27323929)
> RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency.

You got that backwards, chief. RAM latencies have increased tremendously over the last 15 years.

Re:Google says Rowhammer attacks are gaining range as RAM is... by namibj on Saturday May 29, 2021 @05:29AM (#27324080)
Actual latency has improved, but clock-relative latency hasn't. So what?
Re:Google says Rowhammer attacks are gaining range as RAM is... by beebeepka on Saturday May 29, 2021 @05:49AM (#27324148)
No, actual latencies have been climbing since ddr2. Thanks for the downvote though
Re:Google says Rowhammer attacks are gaining range as RAM is... by njibhu on Saturday May 29, 2021 @06:42AM (#27324339)
This article from Crucial probably explains the two arguments: https://www.crucial.com/articles/about-memory/difference-bet... But their conclusion is that practical latency has indeed decreased over the last 15 years.
Re:Google says Rowhammer attacks are gaining range as RAM is... by beebeepka on Saturday May 29, 2021 @08:48AM (#27324910)
After looking at the table, I am ready to swallow my pride. Looks like practical latencies have changed a tiny bit. So me and the person I replied to originally were both wrong
Re:Google says Rowhammer attacks are gaining range as RAM is... by formerly_proven on Saturday May 29, 2021 @12:41PM (#27326415)
You were wrong about the memory latency¹ increasing, but the memory latency² has increased (substantially) for most systems, while the memory latency³ has indeed decreased. In general, the memory latency¹ has not improved much. Of course, the memory latency⁴ has greatly increased, due to the clock frequency being increased so much to enable the higher bandwidth, while the memory latency¹ stayed mostly the same.

[1] as in: access latency of the DRAM [2] as in: how long does a CPU memory read which is _not_ cached take [3] as in: how long does a CPU memory read on average take [4] as in: CL

Re:Google says Rowhammer attacks are gaining range as RAM is... by namibj on Saturday May 29, 2021 @10:14AM (#27325367)
Also note that practical non-high-capacity DDR4 setups reach 3600 CL16. That's way faster than the 3200 CL22 they gave at the bottom.
Re:Google says Rowhammer attacks are gaining range as RAM is... by staticassertion on Saturday May 29, 2021 @04:05PM (#27327883)
No, I was right. Not that it matters, feel free to replace latency with bandwidth. Or replace it with nothing and focus on capacity. It's a blip on my point, which is that the architecture has changed considerably to allow for progress in other areas.
Re:Google says Rowhammer attacks are gaining range as RAM is... by Dylan16807 on Saturday May 29, 2021 @02:05AM (#27323373)
> I'm sure that either it or phenomenon like it are the causes of a lot of odd "glitchy" behaviour that people encounter, because systems are run so close to their limits that there are bound to be cases when they surpass the limits.

I don't know about that. Rowhammer really emphasizes the hammer. A normal read pattern will basically never hit the lines in a small area that much, it will hit cache instead.

> Now you've turned specific access patterns into "attack attempts", and by discriminating against them as such, entirely destroyed a fundamental part of general-purpose computing.

Can you show me any code that causes this kind of access pattern that wasn't deliberately designed to implement rowhammer?

> Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance?

We could go the other way too and glue together multiple SLC cells to increase endurance and cost. Are you claiming there is a specific correct amount of endurance? Because "as much as possible" isn't feasible and "as much as SLC used to have" could only be correct from an extreme coincidence or from conflating nostalgia with correctness. Is there a different number you have in mind?

> This problem didn't exist 15 years ago. It only got created because the RAM companies have forsaken correctness, and somehow managed to convince everyone else that it's not their own fault.

Old ram had errors too. RAM companies can and should put in mechanisms to prevent rowhammer, but if I was buying a stick of ram and had a choice between ECC and perfect rowhammer prevention I'd choose ECC.

Re:Google says Rowhammer attacks are gaining range as RAM is... by metiscus on Saturday May 29, 2021 @03:44AM (#27323696)
What if the choice was between a normal stick of ram and a row hammer mitigated stick with double the CAS timing and that cost 50% more? I guess my point is, considering how fundamental to the design and physics of ram, it is unlikely any product designed to mitigate the problem would be equally performant or economic, at least in a medium timeframe.
Re:Google says Rowhammer attacks are gaining range as RAM is... by Dylan16807 on Saturday May 29, 2021 @04:22AM (#27323832)
I'm much more optimistic. Use target row refresh but do it right. Very low silicon cost, low performance cost under malicious loads, zero performance cost under non-malicious loads.
Re:Google says Rowhammer attacks are gaining range as RAM is... by marcan_42 on Saturday May 29, 2021 @01:09AM (#27323187)
> ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation.

No, ECC is there to correct any errors, including normal errors that occur during operation. It allows us to build technologies that have a much higher raw channel error rate, and decrease the system error rate to better than before. In return, we get much higher density, performance, or whatever other metric you want to optimize for.

This is basic information theory. It is extremely inefficient to attempt to build a channel with a low enough raw bit error rate to be usable. Instead we let the bit error rate rise, then correct the errors with error correction codes. This reduces the bandwidth, proportional to error rate for advanced error correction, which is still a negligible amount compared to the total channel bandwidth - the performance improvements from allowing the raw error rate to rise more than make up for this, and result in orders of magnitude better performance.

> Now you've turned specific access patterns into "attack attempts"

Ever heard of files you can't burn on CD-R, or transmit through certain variants of Ethernet? We use technologies that are probabilistic all the time; they assume data is not correlated with a pseudorandom internal characteristic of some sort. These files only exist because someone explicitly targeted them; the chances of someone stumbling onto such data randomly are basically nonexistent (for any well designed system).

That said, this can be mitigated without throwing ECC out the window.

> RAM should always hold what was last written to it, under all conditions of software accesses.

And it will, after ECC. Why are you drawing an arbitrary line for RAM and saying it can't use ECC? Are you also against HDDs (LDPC error correction), Wi-Fi (convolutional codes), NAND storage of all kinds (increasingly advanced ECC), USB (error detection and re-transmission), TCP (error detection and re-transmission), the internet as a whole (flow control by dropping packets), CDs (three layers of error correction for data!), Bluetooth (FEC and/or ARQ), Digital TV (ATSC, DVB, ISDB all use FEC of some kind), QR codes (reed-solomon), and even things like RAID6? The people who designed these systems aren't dumb, they know the only way to get performance and reliability is to use error correction.

It's a damn miracle we get away with non-ECC RAM, today. That stuff should've died 20 years ago, and ECC should be standard. Even the caches inside CPUs often have ECC these days.

> Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance? That's a rant for another thread...

Yes, manufacturers having a race for the bottom to achieve the highest capacity at the expense of all reliability and performance metrics is a completely different problem that doesn't mean we need to throw ECC out the window, which would also hurt all of those metrics. Optimizing for endurance and performance still requires ECC.

The solution is simple: don't buy QLC flash. That stuff's garbage, at least today.

Or would you prefer to go back to the era of 16MB CompactFlash cards (that's MB) using SLC NOR Flash? That stuff doesn't need ECC, certainly, and the endurance is amazing. Let me know how the cost per gigabyte and write performance metrics work out in your NOR world, though :-)

Or perhaps you'd rather have us store all of our data on 16KB EEPROM chips, with even better endurance! I'm sure nobody will mind the 1KB/s write performance (if you're lucky). Maybe if you RAID a few thousand of those it'll work! Oh wait, then random errors will kill your data anyway, without something like RAID-6. Oops!

> This problem didn't exist 15 years ago.

15 years ago the average computer had 256MB of RAM. I had one of those. And I had a bad RAM bit back then, which I had to mask out with a kernel patch (that I had to port from x86 to amd64, as I was an early adopter of 64-bit), which would've been a non-issue with ECC RAM.

Re:Google says Rowhammer attacks are gaining range as RAM is... by userbinator on Saturday May 29, 2021 @06:49PM (#27328898)
Ever heard of files you can't burn on CD-R

I know about weak sectors. Only shitty (i.e. defective by design) drives wouldn't be able to handle them, while others would do it just fine.

Why are you drawing an arbitrary line for RAM and saying it can't use ECC?

I'm not saying that ECC is not a good idea. I'm saying that ECC errors in RAM occurring simply because of specific access patterns is far from "normal operation". In fact, if you use ECC RAM and see a lot of errors, it should be considered a sign of failing hardware. In other words, advocating for ECC in this scenario is deflecting the blame and compensating for a defective product.

The solution is simple: don't buy QLC flash. That stuff's garbage, at least today

"QLC" and "TLC" are complete misnomers developed specifically to hide the truth. It's not 4 or 3 "levels", but 16 and 8, respectively. 16LC and 8LC are the technically correct terms.

Corporate greed has somehow resulted in everyone parroting the dominant industry position and using Intel and ECC as a scapegoat. That just shows how entrenched their propaganda is.

Re:Google says Rowhammer attacks are gaining range as RAM is... by marcan_42 on Saturday May 29, 2021 @10:42PM (#27330178)
> "QLC" and "TLC" are complete misnomers developed specifically to hide the truth.

I know what those terms mean. I find it amusing that you'd think I don't, given I just rattled off the ECC types used by a dozen technologies and mentioned 3 different kinds of nonvolatile charge trap memory.

Re:Google says Rowhammer attacks are gaining range as RAM is... by 6ue7nNMEEbHcM on Saturday May 29, 2021 @03:04AM (#27323561)
Thanks for posting this. I think there isn't much knowledge about ecc among typical customers. I wish at some point ecc would be the norm. The cost doesn't seem high but for any professional PC usage it seems it's necessary.