I used to work in DRAM manufacturing. I believe 100% in bitflips -- we saw them in testing all the time. I also believe the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. It's really easy to manufacture a weak cell when you're making billions of tiny capacitors at the same time. SRAM might also see lots of bitflips, but that seems even more likely than DRAM to be driven by manufacturing defects, given the greater complexity of the SRAM cell.

>the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects.

back in USSR/Russia gamma rays and aliens were not-an-issue compare to the extremely low reliability of USSR/Russia hardware. The military hardware back then (and some telco hardware built in Russia in 199x (some even for export into Western countries!)) was built as triplicate systems - i.e. primitive quorum/consensus computing.

Similarly, as far as i heard back then, due to low reliability of Itanium especially in the beginning, the Itanium based Tandems were also available as "Tridems".

Except for the case in the USSR when gamma rays were an issue at a train depot computer system due to radioactive cows from near Chernobyl: http://www.jakepoz.com/debugging-behind-the-iron-curtain/

I maintain a tool that reports errors on Linux systems with ECC memory, and also a website describing the tool (http://mcelog.org). I wrote about patterns in the access logs correlated to time some time ago in my blog:

http://halobates.de/blog/p/181

> I wonder if it’s possible to detect solar flares in these logs. Need to find a good data source for them. Are there any other events generating lots of radiation that may affect servers?

Cosmic ray showers [1] potentially. My prediction is that there should be "clusters" of bit errors every few minutes or so (given sufficient amount of servers...).

I was talking to a sysadmin at a local company running a few thousand servers about getting their ECC error logs in order to look for these, but scraping them apparently wasn't trivially managable for them.

[1] - https://en.wikipedia.org/wiki/Air_shower_(physics)

Bit flips in network surely is many orders of magnitude worse than in cache/memory bit flips? And I doubt it has to do with gamma rays.

Those with poor or long distance DSL do see a big increase in errors.

And while hopefully getting rarer now home routers could easily overheat and introduce errors as soon as they saw more than usual traffic.

We recently had issues with the power supply for our router and it also caused issued.

> Bit flips in network surely is many orders of magnitude worse

Yes, but because it's so prevalent that people expect it, they've added checksums on multiple levels, so the network actually performs better. In this particular instance (DNS queries), it's very unlikely that the data was corrupted in transit: "We believe that UDP checksums are effective at preventing 'bitsquat' attacks and other types of errors that occur after a DNS query leaves a DNS resolver and enters the network."

That doesn't take into consideration of flaky home-routers (note that they do recalculate the checksum). So the checksum might be processed on bad data.

For a bad connection it will also be quite common that the checksum will happen to be valid. TCP/UDP checksums are quite bad.

One thing to keep in mind is that the network protocols usually use quite weak and similar checksums. To the extent that there are layer combinations that reliably produce errors that are not detectable by TCP checksum.

Has anyone published results on the impact of large scale TLS deployment on reliability? I’ve seen some older papers on errors which TCP missed but it seems like making a hard fail would be a nice win as the crypto overhead keeps getting cheaper.

I saw them at multiple places in the stack. Usually the Host header showed my domain which rules out the network. Rarely would the Host header be the original domain though that doesn't narrow it down to the network.

The end of gp’s Article points to some work by a verisign engineer that put the network probability very low. So my interpretation was these but flips were happening on device before the network stack was dispatched at a url.

So Ethernet is protected by CRC which is quite good, but it is stripped at both ends and I would not trust the routing chips at all. Some core routers used low reliability FPGAs for example..

If you require data integrity you cannot rely on ethernet's FCS. Personally I trust the core router far more than I trust the cheaper stuff around it, particularly the stuff my ISP gave me for 'free'.

I was talking to a guy at Github who had to debug an enterprise deployment that just stopped working one day.

There was a bit that flipped and apparently took the whole thing to its knees.

This is the stuff nightmares are made of.

The youtube video[1] of his defcon talk is super interesting -although some of his concerns about serving malicious scripts from bitsquatted domains are thankfully mitigated by widespread adoption of TLS.

[1] http://www.youtube.com/watch?v=lZ8s1JwtNas

From your link,

> Bitsquat traffic represents a slice of normal traffic

Wow. That's pretty amazing. With enough analysis, you could (possibly) recover the traffic pattern for major websites. That alone seems to have a lot of interesting implications.

Did also that with facebook.fr/google.fr/akamai.net domains. Got a _lot_ of hits, with mostly mobile user agents.

Was fun; never published my results though..

Same here. If you want a lot of hits, pick their cdn domains.

Get a tls cert and you might even be able to serve the original file with some additional js of your own :)

bitflips are also single char typos in ascii. I'm guessing that is what you saw

This is easily testable because:

1. Not all single char typos are bitflips and when you test it statistically you get more traffic for bitflips than typos. It's a bit more complicated, some typos are really common because they're close to the key.

2. You can look at the raw traffic that comes in and tell what is a simple get request with a fat finger (GET https://en.7ikipedia.org/ vs GET https://en.wikipedia.org/w/index.php?search=america&title=Sp...).

Source: I have some domains that are bitsquats of some high traffic domains. I get access token headers when I do a raw dump of the traffic. (I turned off the servers after satisfying my curiosity.)

Want to stop randos from getting some tiny portion of your traffic's access tokens? Use client side keys and send signed / encrypted access tokens or even full requests.

Yeah I know. It's so far down the list it doesn't matter. But every combination of github.com, microsoft.com, etc is bitsquatted, so this attack can be considered ongoing. This is doubly true if you're on a domain like .co.uk which is a stupid subdomain of .uk and now that .uk addresses are available someone can buy ko.uk and bitsquat the entire fucking country.

An interesting control would have been to flip a single bit but also flip a single char so that it flips several bits and compare the results.

edit: Reading the linked article, they describe one of their validation method: > The Host header contains the domain the HTTP client resolved to connect to the bitsquat server. If the Host header matches the original domain, the corruption occurred on the red path (DNS path). If the Host header matches a bitsquat domain, the corruption occurred on the blue path (content path).

I think the article gives good reasons to believe they are not typos, why do you think otherwise?

While i like the sibling comment's network errors answer better then mine I didn't see any reasoning other then a unfounded statement[1] in the article or on a quick skim for "typo" in the referenced whitepaper:

[1] "These requests were not typos or other manually entered URLs"

Here's a better quote: "There are many frequently resolved domains that few people type or know."

There's a lot more details in the article.

Not to mention, the number of people typing "fbcdn.net" into a browser has to be vanishingly small to begin with.

Is that why some Microsoft download URL with a GUID in it made it to his server? Typo in Windows Update?

I don't think that sounds right.

None of them were typed in. I was bitsquatting a CDN that no one would type into the browser bar. Everything was some resource referenced from a page. If you checked the original page, the correct URL was present in that page.