News for hackers. Stuff that matters.
Re:S3 glitch causes Tarsnap outage by lsc on Friday September 17, 2010 @05:23PM (#1702884)
You'd know much more about these sorts of things than I would, but most filesystems largely ignore disk corruption. If amazon did that, we'd be seeing a lot more trouble than we have. This is where I'm seeing the complexity; detecting and recovering from subtle errors. I've gone a short way down that path and from where I stand it looks quite hairy.

Besides, I've seen data loss caused by ext3 on my own systems that was not caused by disk corruption, so really, at s3's scale, it's got to function (and seems to have been functioning) much better than a regular filesystem.

Re:S3 glitch causes Tarsnap outage by cperciva on Friday September 17, 2010 @05:34PM (#1702912)
S3 has an advantage over regular filesystems there too. S3 doesn't need to provide single-node durability as long as data loss events on different nodes are uncorrelated; so they can protect data with cryptographic checksums and throw it away (from an individual node) at the first sign of corruption, knowing that they'll still have it on all the other nodes.