News for hackers. Stuff that matters.
Re:The mysterious case of the Linux Page Table Isolation pat... by userbinator on Monday January 01, 2018 @12:15PM (#16046891)
IMHO, with RowHammer, the hardware is broken and it will continue to be broken until users complain enough --- maybe to the point of absolutely refusing to buy --- that the manufacturers and designers stop thinking "works 99.9999999999% of the time" is good enough: https://news.ycombinator.com/item?id=12410274
Re:The mysterious case of the Linux Page Table Isolation pat... by avian on Monday January 01, 2018 @12:33PM (#16046980)
Why would users refuse to buy hardware that works 99.9999999999% of the time when they apparently have no problem buying software that works 99% of the time?

Radioactive decays and cosmic particles flipping bits give an upper bound for reliability. You are not going to see low-background packages and rad-hard chips in your iPhone.

Re:The mysterious case of the Linux Page Table Isolation pat... by userbinator on Monday January 01, 2018 @12:42PM (#16047026)
Radioactive decays and cosmic particles flipping bits give an upper bound for reliability well below 99.9999999999%

If it works 99.9999999999%, then it has a failure rate of 0.0000000001%, or 1E-12. Considering that a modern CPU executes approximately 1E9 operations per second, and that regular HDDs have a worse-case BER of 1 in 1E14 bits, 1E-12 is actually rather horrible and the actual error rate of computer hardware is much better than that.

Imagine if a CPU calculated 1+1=3 every 1E12 instructions. At current clock rates, that's a fraction of an hour. Computers simply would not work if CPUs had such an error rate.

I picked the 1E-12 number arbitrarily, but it's quite illustrative of the reliability computers are expected to have, despite their flaws.

Re:The mysterious case of the Linux Page Table Isolation pat... by avian on Monday January 01, 2018 @01:26PM (#16047241)
I edited my reply to remove the actual number. The argument still stands that you cannot expect 100% reliability.
Re:The mysterious case of the Linux Page Table Isolation pat... by blattimwind on Monday January 01, 2018 @02:50PM (#16047676)
Generally speaking, a piece of JavaScript in some 0x0 pixel iframe in a tab you're not even looking at can't summon cosmic particles to manipulate your computer's main memory. Rowhammer can.
Re:The mysterious case of the Linux Page Table Isolation pat... by monochromatic on Monday January 01, 2018 @02:02PM (#16047440)
A small probability of random errors can be lived with. Errors that can be triggered and exploited are unacceptable.
Re:The mysterious case of the Linux Page Table Isolation pat... by brians on Monday January 01, 2018 @04:19PM (#16048111)
No, but it’s straightforward engineering to prevent many of these problems. Error-correcting codes have been well understood for most of a century. Yes, it costs in performance and in money. For life-safety applications, it shouldn’t be optional.

Amazon is hosting life-safety applications in EC2. Commodity x86 hardware is grossly negligent for that environment.

Re:The mysterious case of the Linux Page Table Isolation pat... by jchw on Monday January 01, 2018 @07:55PM (#16049020)
ECC as it is today may not be sufficient. It's mostly useful against single bit flips. Probably better than nothing, though...
Re:The mysterious case of the Linux Page Table Isolation pat... by brians on Monday January 01, 2018 @10:43PM (#16049709)
ECC alone is absolutely insufficient. But ECC can be part of a system design that includes active monitoring and response. I’d expect that system design to also include measurement of ECC events under ordinary conditions, regular re-measurement, and funding for an analysis of the changes and explanation of the difference—just like you’d find in safety engineering in a coal plant, an MRI machine, any sort of engineering that has a professional scientist or engineer on site supervising all operations.

Of course, you’ll also find a tendency there towards specified hardware. They bend or break it to use COTS x86 machines, but—as I think I heard from a comment here last week—nearly nobody ever specified wanting AMT in the initial design, so it’s pretty weird that we’re all buying and deploying it.

Re:The mysterious case of the Linux Page Table Isolation pat... by emilfihlman on Wednesday January 03, 2018 @12:02PM (#16062688)
This is not how probabilities work but eh.
Re:The mysterious case of the Linux Page Table Isolation pat... by tzs on Monday January 01, 2018 @01:32PM (#16047273)
Almost everything I've seen on error rates from radioactive decay and cosmic particles has been on servers in data centers.

I wonder if home systems are equally vulnerable, or if there is something about data center system design or facilities that make them more susceptible?

I ask because I had a couple of home desktop Linux boxes once, without ECC RAM, that were running as lightly loaded servers. I ran a background process on both that just allocated a big memory buffer, wrote a pattern into it, and then cycled through it verifying that the pattern was still there.

Based on the error rates I'd seen published, I expect to see a few flipped bits over the year (if I recall correctly) that I ran these, but I didn't catch a single one.

Later, I bought a 2008 Mac Pro for home, and 2009 Mac Pro for work (I didn't like the PC the office supplied), and used both of those to mid 2017. They had ECC memory, and I never saw any report when I checked memory status that they had ever actually had to correct anything.

So...what's the deal here? What do I need to do to see a bit flip from radiocative decay or cosmic rays on my own computer?

Re:The mysterious case of the Linux Page Table Isolation pat... by asveikau on Monday January 01, 2018 @02:57PM (#16047705)
I think it's multiplication. The odds are low but the number of potential instances is larger. Data centers have larger numbers of machines and those machines are doing repeated work where you observe the result.

Personal machines are typically limited by what your senses can handle. There are few of them for starters. They idle a lot. If many pieces failed inexplicably it's not likely to be something you are personally paying attention to with your senses.

(I have personally observed ram and disk failures on personal machines anyway. And I have seen stuff in my dmesg indicating hardware faults on my personal desktops, but rarely in a way that I notice in actual use not looking at dmesg.)

Re:The mysterious case of the Linux Page Table Isolation pat... by blattimwind on Monday January 01, 2018 @02:54PM (#16047690)
> I wonder if home systems are equally vulnerable, or if there is something about data center system design or facilities that make them more susceptible?

I was told once that today's concrete has a much higher background radiation than brick and mortar from before the 50s. There is also more steel in data centres.

However, I'm not at all sure if background radiation of building materials is even in the right order of magnitude to matter here. Probably not.

Re:The mysterious case of the Linux Page Table Isolation pat... by mikeash on Tuesday January 02, 2018 @11:21AM (#16052782)
A bit of a tangent, but somewhat related to your bit about concrete: steel salvaged from ships built before 1945 is less radioactive than modern steel and is useful for devices that are extremely sensitive to radiation: https://en.wikipedia.org/wiki/Low-background_steel

The reason the modern stuff is more radioactive is the massive number of atmospheric nuclear weapons tests conducted starting in 1945. I imagine the concrete has the same issue.

Re:The mysterious case of the Linux Page Table Isolation pat... by pkaye on Monday January 01, 2018 @03:41PM (#16047925)
I think you need high energy radiation like cosmic rays from space to create problems. So those at higher elevation are at more risk. Heavy material like concrete may block this radiation.
Re:The mysterious case of the Linux Page Table Isolation pat... by ajarmst on Monday January 01, 2018 @05:14PM (#16048335)
Alpha particle emissions are common causes of single-bit errors, especially from ceramic enclosure materials in integrated circuits. Mitigating soft errors from circuit packaging is an active area of research in materials science. Parity bits and CRC error checking are needed precisely to reduce the impact of these errors down to manageable levels.
Re:The mysterious case of the Linux Page Table Isolation pat... by pkaye on Monday January 01, 2018 @08:54PM (#16049270)
"Alpha particles can be stopped by a sheet of paper, beta particles by aluminum, and gamma radiation by a block of lead."

This is what I read online. A typical hardware components is enclosed in a aluminum case so really gamma radiation is of concern right?

Re:The mysterious case of the Linux Page Table Isolation pat... by krallja on Monday January 01, 2018 @09:23PM (#16049387)
GP is saying the radiation is emitted by the component package itself, i.e. the random decay of particles in the ceramic surrounding an IC can cause errors.
Re:The mysterious case of the Linux Page Table Isolation pat... by ajarmst on Monday January 01, 2018 @09:30PM (#16049411)
Lantz, "Soft errors induced by Alpha Particles", 1996. http://ieeexplore.ieee.org/document/510798/?reload=true
Re:The mysterious case of the Linux Page Table Isolation pat... by dboreham on Tuesday January 02, 2018 @12:15AM (#16050031)
It comes from within.
Re:The mysterious case of the Linux Page Table Isolation pat... by emmelaich on Monday January 01, 2018 @07:18PM (#16048876)
Concrete and brick are significant sources of radiation. Especially when new.

I don't know if the sort of radiation they give out is of much risk to computers however.

Re:The mysterious case of the Linux Page Table Isolation pat... by captainmuon on Monday January 01, 2018 @01:24PM (#16047232)
You are not running uniformly random instructions on the CPU. It doesn't matter how many 9s there are in that percentage, if an attacker knows that 0.000...01% code, you have a problem. It actually makes it more insidious, since the chance that is occurs accidentally is basically zero (unlike previous CPU bugs).
Re:The mysterious case of the Linux Page Table Isolation pat... by CamperBob2 on Monday January 01, 2018 @04:39PM (#16048195)
Who cares if it works 99.9999999999999% of the time, if it fails 100% of the time someone deliberately exploits the bug?

This isn't a probability question. I don't understand why it's being discussed as if it were.

Re:The mysterious case of the Linux Page Table Isolation pat... by hujji on Monday January 01, 2018 @12:47PM (#16047047)
The current miniaturization of DRAM circuitry doesn't really allow for a hardware fix for the RowHammer attack. During DRAM manufacturing a test similar to the RowHammer attack exists. This test has certain bounds for passing. If the bounds were tightened up to the level of perfection to prevent the attack it would drop the yield a considerable amount.
Re:The mysterious case of the Linux Page Table Isolation pat... by userbinator on Monday January 01, 2018 @01:01PM (#16047112)
If the bounds were tightened up to the level of perfection to prevent the attack it would drop the yield a considerable amount.

The fact that DRAM older than a few years is effectively immune to RH suggests it is possible to manufacture such. Yes, it will cost more, but I think many would be willing to pay for it like they used to, for none other than the assurance of having more reliable memory.

Re:The mysterious case of the Linux Page Table Isolation pat... by fiddlerwoaroof on Monday January 01, 2018 @01:21PM (#16047225)
Isn’t the problem related to the capacity of current memory modules? So you can have, say, a 32GB module that’s vulnerable or a 8GB module that isn’t? (Assuming the 8GB module uses lower density DRAM chips)
Re:The mysterious case of the Linux Page Table Isolation pat... by HelloNurse on Monday January 01, 2018 @02:12PM (#16047491)
A new 8GB DIMM module has about 1/4 the silicon area of a new 32GB module and the same density (often 1/4 the number of identical chips); only an old 8GB module, made with an entirely different process, would have larger and less dense safer DRAM cells.
Re:The mysterious case of the Linux Page Table Isolation pat... by fiddlerwoaroof on Monday January 01, 2018 @02:38PM (#16047629)
Yeah, that's why I included the qualification
Re:The mysterious case of the Linux Page Table Isolation pat... by joshtopher on Tuesday January 02, 2018 @12:05AM (#16049991)
I only buy DRAM that's aged at least 18 months
Re:The mysterious case of the Linux Page Table Isolation pat... by bloaf on Monday January 01, 2018 @01:09PM (#16047160)
Aren't some manufacturers better than others, though?

https://arstechnica.com/information-technology/2016/03/once-...

Re:The mysterious case of the Linux Page Table Isolation pat... by fulafel on Monday January 01, 2018 @01:45PM (#16047346)
There are many examples of hardware using unreliable underlying layers, countered with whitening/scrambling, redudancy and other data encoding tricks. What would make RowHammer imprevious to these?
Re:The mysterious case of the Linux Page Table Isolation pat... by JoshTriplett on Monday January 01, 2018 @01:55PM (#16047399)
Those tricks cost latency. You can get away with hiding some of that latency in access time to persistent storage. It's much harder to do so with RAM.

ECC helps, and can be done at full rate, but isn't a complete solution for all possible problems. And anything you do in hardware at full RAM speed is expensive.

Re:The mysterious case of the Linux Page Table Isolation pat... by fulafel on Monday January 01, 2018 @02:28PM (#16047569)
This is far away from "hardware fix is impossible" though. In reputable hardware, hardware vendors are expected to maintain correctness in spite of performance advances.

Also, DDR3/4 DRAM is glacially slow in latency terms, it's far from clear that there would be appreciable slowdown. There are already big latency compromises in the standardized JEDEC protocols that are not inherent in DRAM - it would be very two-faced for DRAM vendors to say they'll only trade off latency over backward compatibility or tiny cost savings, but not over correctness.

Re:The mysterious case of the Linux Page Table Isolation pat... by olliej on Monday January 01, 2018 @04:33PM (#16048168)
The trick I heard was to literally throttle the writes if you see repeated parallel writes or similar -- as row hammer depends on rapid writes that should be sufficient. But I also have no real idea how hardware actually works :) (magnets? how do they work?)
Re:The mysterious case of the Linux Page Table Isolation pat... by Dylan16807 on Tuesday January 02, 2018 @06:03AM (#16051179)
You don't need to go that far at all. Just refresh neighbors occasionally when there are tons of accesses to the same place.
Re:The mysterious case of the Linux Page Table Isolation pat... by rasz on Tuesday January 02, 2018 @03:50PM (#16055490)
You dont have to overbuild silicon with huge margins, you can build rowhammer detection mechanism and force refresh when it triggers, thats what target row refresh is doing. Problem is Jedec didnt even bother enforcing it, instead making it optional :/
Re:The mysterious case of the Linux Page Table Isolation pat... by brokensegue on Monday January 01, 2018 @01:20PM (#16047217)
maybe add more bits of ECC?
Re:The mysterious case of the Linux Page Table Isolation pat... by kps on Monday January 01, 2018 @01:26PM (#16047242)
Or indeed any bits of ECC. The fact that Intel omits or disables it on most CPUs is shameful.
Re:The mysterious case of the Linux Page Table Isolation pat... by brokensegue on Monday January 01, 2018 @02:02PM (#16047438)
one bit of ECC isn't sufficient...
Re:The mysterious case of the Linux Page Table Isolation pat... by kps on Tuesday January 02, 2018 @11:40AM (#16053002)
Best kind of correct… but modern wide memory controllers turned (9,8), which only detects one error, into (72,64), which is SECDED, “for free”.
Re:The mysterious case of the Linux Page Table Isolation pat... by brokensegue on Tuesday January 02, 2018 @01:39PM (#16054129)
that is still not sufficient
Re:The mysterious case of the Linux Page Table Isolation pat... by dboreham on Monday January 01, 2018 @01:31PM (#16047269)
There's ECC inside the chips.
Re:The mysterious case of the Linux Page Table Isolation pat... by Symmetry on Monday January 01, 2018 @02:50PM (#16047679)
The internal caches have ECC but rowhammer targets the DIMMs. ECC DIMMs would solve the issue but Intel, for market segmentation reasons, fuses off the ability to handle ECC memory in consumer chips.
Re:The mysterious case of the Linux Page Table Isolation pat... by baq on Tuesday January 02, 2018 @02:51AM (#16050531)
ECC doesn't protect against rowhammer though. It makes it more difficult but not impossible.
Re:The mysterious case of the Linux Page Table Isolation pat... by Dylan16807 on Tuesday January 02, 2018 @06:07AM (#16051191)
If you shut down whatever is triggering errors instead of completely ignoring them, the difficulty level is pretty close to "impossible".
Re:The mysterious case of the Linux Page Table Isolation pat... by zymhan on Monday January 01, 2018 @01:40PM (#16047319)
There is no way the average technology user is going to understand the architectural distinctions of DRAM implementations enough to base their purchasing decisions on that.
Re:The mysterious case of the Linux Page Table Isolation pat... by userbinator on Monday January 01, 2018 @02:01PM (#16047432)
I agree, there needs to be an easy but also reliable way to show the problem. The FDIV bug received much public attention because it could be easily reproduced on Windows' built-in calculator.
Re:The mysterious case of the Linux Page Table Isolation pat... by foota on Monday January 01, 2018 @02:48PM (#16047670)
Aren't there now hardware fixes for this in current generation ram?
Re:The mysterious case of the Linux Page Table Isolation pat... by rphlx on Monday January 01, 2018 @07:43PM (#16048969)
The DRAM mfgs did respond to some extent & it is much harder - though perhaps not entirely impossible - to RH DDR4.
Re:The mysterious case of the Linux Page Table Isolation pat... by rasz on Tuesday January 02, 2018 @03:56PM (#16055547)
optional = forget about it
Re:The mysterious case of the Linux Page Table Isolation pat... by jtchang on Monday January 01, 2018 @02:01PM (#16047435)
Umm this is very shortsighted and absolutely trash.

There are many types of users and some don't care if it fails in a blue moon. Hell 99% might be good enough if the price was right.

Re:The mysterious case of the Linux Page Table Isolation pat... by blattimwind on Monday January 01, 2018 @02:55PM (#16047696)
99% would basically be the region where the CPU itself can't power on any more because it is too unreliable to execute any kind of power-on sequence.
Re:The mysterious case of the Linux Page Table Isolation pat... by dang on Monday January 01, 2018 @03:55PM (#16047992)
Please don't call names in arguments here. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html