Someone correct me if I'm wrong but to my understanding FB uses Btrfs in either RAID 0, 1, or 10 only and not any of the parity options.

RAID56 under Btrfs has some caveats but I'm not aware of any annecdata (or perhaps I'm just not searching hard enough) within the past few weeks or months about data loss when those caveats are taken under consideration.

> RAID56 under Btrfs has some caveats but I'm not aware of any annecdata (or perhaps I'm just not searching hard enough) within the past few weeks or months about data loss when those caveats are taken under consideration.

Yeah this is something that makes me consider trying raid56 on it. Though I don't have enough drives to dump my current data while re-making the array :D (perhaps this can be changed on the fly?)

What's your starting array look like? If you're already on Btrfs then I recall you could do something like `btrfs balance -d raid6 -m raid1c3 /`

https://btrfs.readthedocs.io/en/latest/Balance.html

Yeah I'm on btrfs raid 1 currently, with 1x1TB + 2x3TB + 2x4TB drives. Gotta love btrfs's flexibility regarding drive size :D

I'll have a look, thanks! I guess failing this will make me test by backup strategy that I have never tested in the past.

Out of curiosity, how much total storage do you get with that drive configuration? I've never tried "bundle of disks" mode with any file system because it's difficult to reason about how much disk space you end up with and what guarantees you have (although raid 1 should be straightforward, I suppose).

I get half of the raw capacity, so 7.5TB. Well a bit less due to metadata, 7.3TB as reported by df (6.9TiB).

For btrfs specifically there is an online calculator [1] that shows you the effective capacity for any arbitrary configuration. I use it whenever I add a drive to check whether it’s actually useful.

1: https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&...

I get half of the raw capacity, so 7.5TB. Well a bit less due to metadata, 7.3TB as reported by df (6.9TiB).

For btrfs specifically there is an online calculator [1] that shows you the effective capacity for any arbitrary configuration. I use it whenever I add a drive to check whether it’s actually useful.

1: https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&...

Here are raw outputs:

$ sudo btrfs fi df /media/storage� Data, RAID1: total=6.34TiB, used=6.22TiB System, RAID1: total=32.00MiB, used=992.00KiB Metadata, RAID1: total=11.00GiB, used=8.71GiB GlobalReserve, single: total=512.00MiB, used=0.00B $ sudo btrfs fi us /media/storage� Overall: Device size: 13.64TiB Device allocated: 12.70TiB Device unallocated: 966.54GiB Device missing: 0.00B Device slack: 0.00B Used: 12.47TiB Free (estimated): 600.31GiB (min: 600.31GiB) Free (statfs, df): 547.55GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no

Data,RAID1: Size:6.34TiB, Used:6.22TiB (98.20%) /dev/sdc 2.53TiB /dev/sdk 2.54TiB /dev/sdf 837.00GiB /dev/sdb 3.40TiB /dev/sdi 3.39TiB

Metadata,RAID1: Size:11.00GiB, Used:8.71GiB (79.22%) /dev/sdc 3.00GiB /dev/sdk 4.00GiB /dev/sdf 2.00GiB /dev/sdb 6.00GiB /dev/sdi 7.00GiB

System,RAID1: Size:32.00MiB, Used:992.00KiB (3.03%) /dev/sdc 32.00MiB /dev/sdk 32.00MiB

Unallocated: /dev/sdc 198.49GiB /dev/sdk 187.49GiB /dev/sdf 92.51GiB /dev/sdb 243.02GiB /dev/sdi 245.02GiB

I get half of the raw capacity, so 7.5TB. Well a bit less due to metadata, 7.3TB as reported by df (6.9TiB).

For btrfs specifically there is an online calculator [1] that shows you the effective capacity for any arbitrary configuration. I use it whenever I add a drive to check whether it’s actually useful.

1: https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&...

Here are raw outputs:

$ sudo btrfs fi df /media/storage� Data, RAID1: total=6.34TiB, used=6.22TiB System, RAID1: total=32.00MiB, used=992.00KiB Metadata, RAID1: total=11.00GiB, used=8.71GiB GlobalReserve, single: total=512.00MiB, used=0.00B $ sudo btrfs fi us /media/storage� Overall: Device size: 13.64TiB Device allocated: 12.70TiB Device unallocated: 966.54GiB Device missing: 0.00B Device slack: 0.00B Used: 12.47TiB Free (estimated): 600.31GiB (min: 600.31GiB) Free (statfs, df): 547.55GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no

Data,RAID1: Size:6.34TiB, Used:6.22TiB (98.20%) /dev/sdc 2.53TiB /dev/sdk 2.54TiB /dev/sdf 837.00GiB /dev/sdb 3.40TiB /dev/sdi 3.39TiB

Metadata,RAID1: Size:11.00GiB, Used:8.71GiB (79.22%) /dev/sdc 3.00GiB /dev/sdk 4.00GiB /dev/sdf 2.00GiB /dev/sdb 6.00GiB /dev/sdi 7.00GiB

System,RAID1: Size:32.00MiB, Used:992.00KiB (3.03%) /dev/sdc 32.00MiB /dev/sdk 32.00MiB

Unallocated: /dev/sdc 198.49GiB /dev/sdk 187.49GiB /dev/sdf 92.51GiB /dev/sdb 243.02GiB /dev/sdi 245.02GiB

Just want to do a follow up and make a correction that the command to go from whatever to RAID 6 for data and RAID 1c3 for metadata in Btrfs is instead: `btrfs balance -dconvert=raid6 -mconvert=raid1c3 /` instead of what I originally posted

> It's funny because Facebook uses btrfs for their systems & doesn't have these issues.

they likely have distributed layer on top which takes care of data corruption and losses happening on specific server