News for hackers. Stuff that matters.
Re:135TB for $7,384 - Backblaze Pod 2.0 by brianwski on Wednesday July 20, 2011 @01:54PM (#2786695)
Yes, the RAM is ECC. Here is a link: http://www.crucial.com/store/partspecs.aspx?imodule=CT25672B...

The ECC RAM absolutely does find and corrects problems (we see them in the logs). However, just to be absolutely clear we would not need ECC RAM -> Backblaze checksums EVERYTHING on an end-to-end basis (mostly we use SHA-1). This is so important I cannot stress this highly enough, each and every file and portion of file we store has our own checksum on the end, and we use this all over the place. For example, we pass over the data every week or so reading it, recalculating the checksums, and if a single bit has been thrown we heal it up either from our own copies of the data or ask the client to re-transmit that file or part of that file.

At the large amount of data we store, our checksums catch errors at EVERY level - RAM, hard drive, network transmission, everywhere. I suppose consumers just do not notice when a single bit in one of their JPEG photos has been flipped -> one pixel gets every so slightly more red or something. Only one photo changes out of their collection of thousands. But at our crazy numbers of files stored we see it (and fix it) daily.

Re:135TB for $7,384 - Backblaze Pod 2.0 by db48x on Wednesday July 20, 2011 @04:18PM (#2787247)
Merkle Trees FTW, but it sounds like you've basically reimplemented ZFS.

Have you tried using ZFS on one of these?

Re:135TB for $7,384 - Backblaze Pod 2.0 by brianwski on Wednesday July 20, 2011 @05:12PM (#2787440)
Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not available on Linux and we're not really looking to switch to OpenSolaris.

ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet.

By the way, at Backblaze I've felt like we have had to implement several things that I would have guessed would be standard "off the shelf". One example: When a customer wants to download a restore file with a web browser, are you aware there are no checksums for over-the-network transfers other than the built in (completely unacceptable) 16-bit TCP checksum? You are virtually guaranteed to have an undetected corruption within 30 Gbytes of download, which is basically what we like to call "a totally average customer restore". So Backblaze had to write our own custom reliable, restartable "downloader". It boggles my mind that the whole internet is throwing undetected errors on HTTP downloads and nobody cares to fix the protocol?!! Where the heck is Google, Apple, Facebook, Microsoft, or <insert standards body> defining a standard for web browser downloads larger than a few GBytes?

Re:135TB for $7,384 - Backblaze Pod 2.0 by db48x on Wednesday July 20, 2011 @05:41PM (#2787550)
Actually, zfs-on-linux is doing pretty well. It's not production ready yet, of course.

As for HTTP, I agree with you entirely! There is actually a standard that lets you put a hash (SHA-1 or SHA-256 or something) on an http anchor link, and the browser will verify it when the download finishes. It hasn't really gained very wide adoption though. Personally, I think something akin to Bittorrent is a better solution, since it doesn't have to redownload the entire file when it detects an error. It's ironic that our videos often have better data integrity than the graphics driver installer that I just downloaded, or the browser that I downloaded yesterday.

Re:135TB for $7,384 - Backblaze Pod 2.0 by silvestrov on Wednesday July 20, 2011 @05:55PM (#2787584)
The OSX Lion download is +3 GB, so Apple customers must take care of TCP errors (or at least Apple Support would be swamped with support requests).

Two possible ways of solving this: make a download into e.g. 16 MB chunks and append a SHA-1 checksum for each chunk (not enterely unlike BitTorrent). Then re-download the chunks where the SHA-1 doesn't match. Another solution would be to use e.g. Red Solomon error correction.

Re:135TB for $7,384 - Backblaze Pod 2.0 by brianwski on Wednesday July 20, 2011 @06:35PM (#2787690)
Yep, Apple solves this problem by using their "AppStore" application to reliably download the file in chunks (like you suggested). My complaint is that Apple should extend Safari to be able to download the OS X Lion download reliably. (Then let the rest of us use that protocol.) It seems a bit insane that the OS group at Apple had to write their own custom download application, but did not think to extend their web browser to allow for "restartable downloads that are no longer corrupted".

Microsoft does the same thing -> they have a custom application to download their OS updates reliably. They do not use Internet Explorer's regular download capability because of the limitations.

It baffles me that the big browser companies (Google with Chrome, Apple with Safari, Microsoft with IE) are not interested in building a standard reliable downloader. I mean, what else does a web browser really do?

Re:135TB for $7,384 - Backblaze Pod 2.0 by andrewf on Wednesday July 20, 2011 @10:40PM (#2788317)
gzip and zlib HTTP encodings (different containers for "deflate" compression) include a checksum of the uncompressed content (a CRC-32 variant and Adler-32 respectively). If compression isn't wanted, you can wrap uncompressed bytes with a deflate structure - although it'll have to be split into ~64k chunks with a few header bytes ahead of each.

I'm not sure which HTTP client implementations reliably test those checksums, though. (Or whether CRC32/Adler32 would be adequate for 30 gigs!)

Re:135TB for $7,384 - Backblaze Pod 2.0 by mrb on Thursday July 21, 2011 @12:07AM (#2788488)
And when using HTTPS, all corruption introduced by the network stack will be detected (thanks to the MAC in each TLS record).

But brianwski's point remains that neither HTTP encodings, nor HTTPS can detect certain errors like in-memory corruption of the data after it has been read from disk, but before it has been sent out over the HTTP connection.

Re:135TB for $7,384 - Backblaze Pod 2.0 by zerosanity on Wednesday July 20, 2011 @05:32PM (#2787509)
We're using FreeBSD's ZFS support and haven't had any hiccups to complain about.
Re:135TB for $7,384 - Backblaze Pod 2.0 by thaumaturgy on Wednesday July 20, 2011 @07:11PM (#2787796)
We've had a backup server running for a client with FreeBSD's ZFS for about a year now, also with zero hiccups. We first tried to get this kind of stuff working about three or four years ago, and went the OpenSolaris/ZFS route -- what a pain that was. OpenSolaris somehow corrupted its own boot partition one day and just completely refused to start with a totally cryptic error code.

The last time we tried Linux for this, it couldn't do it. Maybe that's gotten better though.

We also tried DragonflyBSD, but it had hardware support issues, and OpenBSD, but unfortunately OpenBSD just simply cannot do large filesystems. At all.

ZFS-on-FreeBSD is the way to go at the moment, I think.

Re:135TB for $7,384 - Backblaze Pod 2.0 by finnh on Friday July 22, 2011 @03:16PM (#2794814)
ZFS on FreeBSD works a charm - I was happy to switch to FreeBSD after having done OpenSolaris & Nexenta specifically for ZFS.

I actually use ZFS on Mac OS, as well, and have been happy with it there (thanks Dustin Sallings!).

Re:135TB for $7,384 - Backblaze Pod 2.0 by brmj on Thursday July 21, 2011 @02:24PM (#2790616)
Solaris isn't your only option. Various BSDs support ZFS quite well.
Re:135TB for $7,384 - Backblaze Pod 2.0 by jbooth on Wednesday July 20, 2011 @03:02PM (#2786963)
Huh, didn't know there was an i3 that supported ECC. Cool.
Re:135TB for $7,384 - Backblaze Pod 2.0 by RShackleford on Wednesday July 20, 2011 @07:45PM (#2787900)
Curious, according to Intel, it the i3 doesn't do ECC: http://www.intel.com/support/processors/corei3/sb/CS-031175....

But the Supermicro board says it can: http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm

The board also supports Xeon processors, so it might be that it only supports ECC with Xeon. However, I didn't see anything in the docs to support this, instead it says "Dual Core processors of Ci3 and Pentium: support ECC UDIMM only"

I'm curious since I ended up springing for Xeon in my system specifically for ECC. Now I'm wondering if I made a mistake...

Re:135TB for $7,384 - Backblaze Pod 2.0 by wmf on Thursday July 21, 2011 @12:44AM (#2788551)
IIRC, some vendors wanted to offer super-low-end dual-core servers but rather than create a dual-core Xeon SKU Intel decided to just not disable ECC on some of the Pentiums and i3s. (As if their branding wasn't f'd enough.) So yeah, i3s support ECC when used in a server board.
Re:135TB for $7,384 - Backblaze Pod 2.0 by brianwski on Thursday July 21, 2011 @04:49PM (#2791224)
We didn't think we even had a choice! From: http://www.supermicro.com/manuals/motherboard/3420/MNL-1130....

"The X8SIL/X8SIL-F/X8SIL-V supports up to 16GB of DDR3 ECC UDIMM or up to 32GB of ECC DDR3 RDIMM (1333/1066/800 MHz in 4 DIMM slots.)"

Below are what we assume are ECC errors from an i3 based Backblaze pod, when this happens it is usually bad RAM (bad enough to crash the pods, replace with fresh RAM repairs it):

### IPMI LOG: ####

ipmitool sel elist | grep -v "Fan FAN" 69 | 01/04/2011 | 20:21:52 | Memory | Correctable ECC | Asserted 73 | 01/04/2011 | 21:15:50 | Memory | Correctable ECC | Asserted