News for hackers. Stuff that matters.
Re:How Mail.Ru Reduced Email Storage From 50 To 32 PB by kevincox on Saturday January 28, 2017 @07:30PM (#13510731)
This is a surprisingly primitive and simple system. A twice replicated file is a huge space overhead for not very reliable storage. (I mean the probability of losing two disks at the same time isn't high, but it's also not immeasurably low)

For example fb talked about how they use erasure coding to use 1.4x space overhead for far higher reliability.

https://code.facebook.com/posts/1433093613662262/-under-the-...

While erasure coding can have performance impact (although mostly only on failures) a multi-tier storage system will eliminate that. I presume that most attachments aren't read more the a week or two after being received.

So while it is definitely nice to get some savings through deduplication there is a lot more gains that can be had by more intelligent storage of the data.

There are no comments on this comment yet.