You can probably achieve what you're looking for by stacking a few filesystems. For example, you could create a separate ZFS pool/vdev with a single full-disk zvol on each disk. Then use mdadm to create a RAID array of the zvols. Then put ext4 (or whatever) on the mdadm array.

I've done something similar for the purpose of getting FDE with ZFS in linux. It can be a little finicky, but it's definitely workable.

One ZFS-specific caveat (which may conflict with your desire to get high storage efficiency): you way need to prevent your ZFS pools from filling up too much [1]. You can either enable discard/TRIM on the whole stack, so the top level FS (e.g. ext4) can let ZFS know when a block is actually free. Or alternative just to limit your zvols to 85% (for example) of their respective pools. The latter is my preference, because there was originally a bug with discard in zfs and it's not immediately clear if it's totally fixed (although my fstrim tests seemed to work out fine).

[1] https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha...

Hmm, that actually sounds workable. I could even format the mdadm device as ZFS too if I really wanted. I am somewhat worried about performance, have you had any issues with that?

I haven't had any issues with performance, but then again my requirement was just "reasonable performance".

I ran some quick benchmarks (data below). Obviously this is far from rigorous, but maybe it'll be useful. In previous tests I found that volblocksize=128K was optimal for my stack -- which is why the last benchmarks use that setting.

Every additional ZFS filesystem in the stack may reduce storage efficiency (minimum free space requirements [1]; metadata & checksum overhead [2][3]) -- that's why I used ext4 as the top layer instead of another ZFS.

[1] (as mentioned before) https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha...

[2] https://news.ycombinator.com/item?id=14756360

[3] https://forums.freenas.org/index.php?threads/what-is-the-exa...

  Test setup:
   debian stable
   kernel 4.9.0-3-amd64
   zfs 0.6.5.9-5
   ZFS "pool": mirror with 2x 7200rpm drives

  Benchmark command:
   for i in `seq 1 10`; do sync; dd if=/dev/zero of=DEST bs=1M count=1024 conv=fdatasync; done

  zfs mirror -> dataset
   Data (MB/s): 125,115,104,135,148,170,135,151,118,119
   Mean (MB/s): 132.0
   Std.dev.: 19.9

  zfs mirror -> zvol (volblocksize=8K [default])
   Data (MB/s): 150,115,127,125,122,118,105,118,124,128
   Mean (MB/s): 123.2
   Std.dev.: 11.6

  zfs mirror -> zvol (volblocksize=128K)
   Data (MB/s): 68.5,112,115,114,94.3,85.1,83.1,98.4,120,108
   Mean (MB/s): 99.8
   Std.dev.: 16.9

  zfs mirror -> zvol (volblocksize=128K) -> luks -> ext4  (my stack)
   Data (MB/s): 130,94.4,109,139,138,125,94.9,124,134,133
   Mean (MB/s): 122.1
   Std.dev.: 16.8

edit: formatting

Can you please elaborate what are the advantages of such configuration?

The advantage is that you can get all the features you want, even though they aren't all available in one filesystem.

In my case, I wanted a reliable filesystem, RAID1/mirror support, block-level checksumming, and full-disk encryption. No filesystem provides these on linux right now. My solution was therefore to use ZFS to provide a mirrored, checksummed, reliable volume -- onto which I put a standard LUKS-encrypted ext4 filesystem. In the past I had tried the opposite (LUKS on the bare drives, then a ZFS mirror of the 2 decrypted volumes), but it was kinda annoying to manage, and I don't really need the other ZFS features (like snapshotting).

In the grandparent post, the requirement was the ability to expand the RAID volume without rebuilding (something that ZFS doesn't offer), plus checksumming and reliability (which ZFS does offer). So one option would be to use mdadm to manage the RAID array, and then put ZFS on the resultant volume in order to get checksumming.

The disadvantages are: extra complexity; extra overhead; more potential points of failure; more management hassle; etc. As soon as encryption for ZFSonlinux is stable, I'll be very happy to drop this filesystem stacking in favor of that!

There kinda isn't such a thing at this point. I sure wish there was! And this is the point behind the article: Wouldn't it be awesome if ZFS had continued developing as the super filesystem it looked like last decade? It would do all this and more! Frankly if ZFS supported re-balancing ("re-RAID-ing") and hybrid pools, it would be pretty much everything we all need. I do hope ZFS or Btrfs or something gets there.

It's so close though. BTRFS and Ceph both have what I want, they're just unstable. The BcacheFS dev has told me he's close but not quite there yet.

Hopefully one of the other solutions will work, otherwise I'll just have to build it.

bcachefs really does seem as if it will be our holy grail, we have very similar needs.

They have a patreon if you wish to contribute that way.

I already contribute and help out with testing and the occasional docs on IRC :)

XFS has gained (some) checksumming and CoW support.

At this rate XFS will end up evolving to add all the features btrfs promised before the latter makes them stable.

> ZFS doesn't support addition of disks unless you're happy to put a RAID0 on top of your RAID5/6 and it doesn't support removal of disks at all when parity is involved. It is possible to migrate by putting giant sparse files on the existing storage, filling the filesystem, removing a sparse file, removing a disk from the original FS and "replacing" the sparse file with the actual disk but this is somewhat risky.

It may be "RAID0 on top of your RAID5/6" but it's all integrated and works smoothly. What I do is have two raidz2 vdevs of 4 disks each, one twice the size of the other, and alternate which one I upgrade (i.e. I started with 4x250gb disks and 4x500gb disks, a few years later replaced the 250gb ones with 1tb ones, then the 500gb ones with 2tb ones, and most recently the 1tb ones with 4tb ones). But yeah I am now stuck with at least 8 disks and if I wanted to migrate off them I'd have to do so all in one go.

> It may be "RAID0 on top of your RAID5/6" but it's all integrated and works smoothly.

I know that it does but I don't like the way you lose your entire pool if a single vdev fails. I'd have less of a problem with it if a pool could distribute data such that a vdev failure results in the loss of only what was on that vdev.

Shrug. I figure losing a random half of my files is pretty much as bad as losing all of them (if I wanted to make an explicit split into two halves I could use two separate pools).

Re: contact sales, I totally hear what you're saying. It's not always a bad idea to talk to a sales person, and a good one can steer you towards the options you'd actually want. But yeah, I generally don't want to talk to a salesperson either.

Yes, but would it kill them to just put a pricing there? Nowadays you can get the list price of sending a Rocket into space[1] but not how much some software is going to cost you

[1] http://www.spacex.com/about/capabilities in case you need one

Have you seen Quantcast File System? http://quantcast.github.io/qfs/

I hadn't and it looks good but it'll only allow 3 recovery stripes or none, which isn't optimal for me.

Have you considered snapraid? It's offline though, but seems to tick all your boxes otherwise.

Yes, I've looked into SnapRAID but I'm not a fan of offline and IIRC I didn't have enough memory to run it last time I tried it out.

For storage spaces try fixed size, you can later enlarge the volumes, faster.

I was using fixed size, it didn't help. Read speeds were okay but write always capped out at 25MB/s.