News for hackers. Stuff that matters.
Re:Dropbox’s Exodus from the Amazon Cloud by james_cowling on Tuesday March 15, 2016 @01:45AM (#11287677)
No filesystem - the rust codebase directly handles the disk layout and scheduling.

Our previous version of the storage layer ran on top of XFS which was probably a bad choice since we ran into a few XFS bugs along the way, but nothing serious.

Re:Dropbox’s Exodus from the Amazon Cloud by jaysoncena on Tuesday March 15, 2016 @02:29AM (#11287857)
this is really cool! Can you give more details on how blocks, replications, metadata, etc. are being handled?
Re:Dropbox’s Exodus from the Amazon Cloud by james_cowling on Tuesday March 15, 2016 @03:26AM (#11288043)
Lazy answer but we'll blog about this in the next month.

On a high level it's variable-sized blocks packed into 1GB extents, which are then aggregated into volumes and erasure coded across a set of disks on different machines/racks/rows/etc. We also replicate cross-country in addition to this. Live writes are written into non-erasure-coded volumes and encoded in the background.

The volume metadata on the disks contains enough information to be self-describing (as a safety precaution), but we also have a two-level index that maps blocks to volumes and volumes to disks.

More info to come later.

Re:Dropbox’s Exodus from the Amazon Cloud by kdkeyser on Tuesday March 15, 2016 @06:20AM (#11288617)
Are live writes replicated in real time, or are they locally staged (which is what Facebook does, I believe)

Also, do you mimic the eventually consistent behaviour of AWS, or do you offer a stronger form of consistency?

Re:Dropbox’s Exodus from the Amazon Cloud by jamwt on Tuesday March 15, 2016 @12:07PM (#11290651)
Live writes are written out with 4x redundancy in the local zone, then asynchronously replicated out to the remote zone. Some time later, it is erasure coded into a more space-efficient format, independently in each zone.