S3QL: an S3 FS with encryption, de-dup, immutable trees and snapshotting

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by pja on Tuesday September 01, 2015 @04:31AM (#10151307)

raid1 on all data. This isn't a backup - it's for end-to-end detection of bit flips in (some of) the storage path (and at rest). This needs a better solution (zfs?)

Yeah, btrfs or zfs are the better choice for this than RAID1 in the modern world I think.

ZFS anywhere it runs natively, btrfs on Linux.

(My experience with btrfs is that it’s fine on a server, but on laptops I’ve ended up with unmountable, unfixable filesystems a number of times. To be fair to btrfs I’ve always been able to recover the data though.)

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by riquito on Tuesday September 01, 2015 @04:51AM (#10151347)

Anecdotal, but I use btrfs on my laptop since august 2014, had a good number of hard power off not related to btrfs and the filesystem never failed me once.

I had some problem with space though using intensively docker: in some situations btrfs thought that the free space was finished and I had to manually rebuild the metadata.

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by vidarh on Tuesday September 01, 2015 @05:22AM (#10151431)

The Docker-on-Btrfs problems are/were severe enough that CoreOS now defaults to overlayfs on Ext4. I have a bunch of EC2 servers on CoreOS, and we ended up wiping Docker partitions regularly until that switch...

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by pja on Tuesday September 01, 2015 @09:43AM (#10152229)

IIRC docker (or any other virtual machine filesystem) in a single image file and hashing filesystems mix really, really badly.

Docker could use BTRFS / ZFS snapshots instead of using whole filesystem image files though, so this ought to get better.

[ Parent | on HN ]