(I don't use btrfs or any other COW filesystem because of significantly worse performance with some kinds of workloads, but it has nothing to do with maturity of any of them.)
I use RAID-Z2 in lots of places for bulk storages purposes (HPC).
There's a reason why Ceph added erasure coding:
* https://ceph.io/en/news/blog/2017/new-luminous-erasure-codin...
* https://docs.ceph.com/en/latest/rados/operations/erasure-cod...
When you're talking about PB of data, storage efficiencies add up.
The people on IRC tend to default to "unless you're using an enterprise drive, it's probably buggy and doesn't respect write barriers", which shouldn't have mattered because there was no system crash involved.
Yes, I did test my RAM, I know it's fine. For comparison, I've (unintentionally) ran a ZFS system with bad RAM for years and it only manifested as an occasional checksum error.
Just luck. Software can't defend itself against bad RAM. There's always the possibility that bad RAM will cause ZFS to corrupt itself in some way it can't recover itself from.
Everything is in RAM. The kernel, the ZFS code, everything. All of that is vulnerable to corruption. No matter how fancy ZFS is, it can't stop its own code from being corrupted. It's just luck that it didn't happen.
Be careful though. If whatever data was to be written got corrupted early enough, ie before ZFS got to see it, it happily wrote corrupted data to disk with matching checksum and you're none the wiser. But yes, it didn't blow up the entire Filesystem unlike btrfs likes to do.
This includes plenty of random power losses.
https://daltondur.st/syno_btrfs_1/
Sorry about that!
Given the rather cheap price of durable storage these days, I would favour rock solid, high quality code for storing my data, at the expense of some optimisations. Then again, I still like RAID, instantaneous snapshots, COW, encryption, xattr, resizable partitions, CRC... It's it possible to have all this with acceptable performance and simple code bricks combined and layered on top of each other?
Or that the complexity is such that if a new bug is found, it may take a long time to be fixed because of the complexity, or it is fixed fast and has unexpected knock-on effects even for circumstances on the common path.
Something that takes a long time to be declared stable/reliable because of its complexity, needs to spend a long time after that declaration without significant issues before I'll actually trust it. Things like btrfs definitely live in this category.
bcachefs even won't be something I use for important storage until it has been battle-tested a bit more for a bit longer, though at this point it is much more likely to take over from my current simple ext4-on-RAID arrangement (and when/if it does, my backups might stay on ext4-on-RAID even longer).
yeah, features rich/complete fs is complicated, that's why we have very few of them.
BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why
Because XFS is far quicker for server-related software such as databases and virtual machines, which are weak points on btrfs due to its COW model.
Do you know how ZFS handles that?
For two devices, 1x redundancy (so 2x copies of everything) will always limit your storage to the size of the smaller device otherwise it is not possible to have two copies of everything you need to store. As soon as you add a third device of at leats 100gb (or replace the 100gb device with one at least 200gb) the other 100gb of your second device will immediately come into play.
Uneven device size support is most useful when: * You have three or more devices. * You want flexibility rwt array growth (support for uneven devices usually (but not always) comes with better support for dynamic array reshaping). * You want better quick repair flexibility: if 4Tb drive fails, you can replace it with 2x2Tb if you don't have a working 4Tb unit on-hand. * You want variable redundancy (support for uneven devices sometimes comes with support for variable redundancy: keeping 3+ copies of important data, or data you want to access fastest via striping of reads, 2 copies of other permanent data, and 1 copy of temporary storage, all in the same array). In this instance the “wasted” part of the 200gb drive in your example could be used for scratch data designated as not needing to be stored with redundancy.
e.g. you have 3 100GB drives, total capacity in raid 1 is 150GB.
If you replace a broken one with a 200GB one, the total capacity will be increased to 200GB.
You can then extend the pool by adding more such partition-based vdevs as you replace disks, just add the new partitions and add new vdevs.
So if you have a 1TB disk, a 2TB disk and a 4 TB disk, you could have mirrors (d1p1,d3p1), (d2p1, d3p2), (d2p2, d3p3) with a total of 3TB mirrored, and 1 TB available. If you swap the 1TB for a 2TB disk, partition it, replacing the old d1p1 partition with the new and resilver, and once that's done you can add the mirror (d1p2, d3p4) and get the full 4TB redundant storage.
Not a great solution though, as it requires a lot of manual work, and especially write performance will suffer because ZFS will treat the vdevs as being separate and issue IOs in parallel to them, overloading the underlying devices.
1. The scheduler doesn't really exist. IIRC it is PID % num disks.
2. The default balancing policy is super basic. (IIRC always write to the disk with the most free space).
3. Erasure coding is still experimental.
4. Replication can only be configured at the FS level. bcachefs can configure this per-file or per-directory.
bcachefs is still early but it shows that it is serious about multi-disk. You can lump any collection of disks together and it mostly does the right thing. It tracks performance of different drives to make requests optimally and balances write to gradually even out the drives (not lasering a newly added disk).
IMHO there is really no comparison. If it wasn't for the fact that bcachefs ate my data I would be using it.
ZFS does something smarter here, it keeps track of the queue length for each drive in a mirror, and picks the one with the lowest number of pending requests.
- I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. I think the point of raid1 is to minimize downtime when losing a device and if you lose the other device before returning it to good state, that's 100% on you.
- Poor management tools compared to md (though bcachefs might be in the same boat). Some tools are poorly thought, e.g. there is a tool for defragmentation, but it undoes sharing (so snapshots and dedupped files get expanded).
- If a drive in raid1 drops but then later comes back, btrfs is still quite happy.
- Need of using btrfs balance, and in a certain way as well: https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-... .
- At least it used to be difficult to recover when your filesystem becomes full. Helps if you have it on LVM volume with extra space.
- Snapshotting or having a clone of a btrfs volume is dangerous (due to the uuid-based volume participant scanning)
- I believe raid5/6 is still experimental?
- I've lost a filesystem to btrfs raid10 (but my backups are good).
- I have also rendered my bcachefs in a state where I could no longer write to the filesystem, but I was still able to read it. So I'm inclined to keep using bcachefs for the time being.
Overall I just have the impression that btrfs was complicated and ended up in a design dead-end, making improvements from hard to difficult, and I hope that bcachefs has made different base designs, making future improvements easier.
Yes, the number of developers for bcachefs is smaller, but frankly as long as it's possible for a project to advance with a single developer, it is going to be the most effective way to go—at the same time I hope this situation improves in the future.
Add "degraded" to default mount options. Solved.
The annoying part of this is that if you do reboot the system, it will never end up responding to a ping, meaning you need to visit the host yourself. In practice it might even have other drives you could configure remotely to replace the broken device. I use md's hot spare support routinely: an extra drive is available, should any of the drives of the independent raids fail.
Granted, md also has decent good monitoring options with mdadm or just cat /proc/mdstat.
How often do people reboot their systems? IMO, if it's running (without going into "read-only filesystem" mode) with X disks, it should boot with the same X disks. Otherwise, it might be running fine for a long time, and it arbitrarily not coming back in case of a power failure (when otherwise it would be running fine) is an unnecessary trap.
Not doing so is especially upsetting when you discover you forgot to flip the setting only when a drive fails with the machine in question several hours' drive away (standalone remote servers like that tend not to have console access).
I think 'refusing to boot' is probably the right default for a workstation, but on the whole I think I'd prefer that to be a default set by the workstation distro installer rather than the filesystem.
Complete Principle of Least Surprise violation given everything else's (that I'm aware of) RAID1 setups will still boot fine.
Also said monitoring should then notify you of an unexpected reboot and/or a dropped out disk, which you can then remediate in a planned fashion.
If this was a new concept then defaulting all the safety knobs to max would seem pretty reasonable to me, but it's an established concept with established uses and expectations - a server distro's installer should not be defaulting to 'cause an unnecessary outage and require unplanned physical maintenance to remediate it.'
Try CachyOS (or at least the ZFS-Kernel) it has excellent ZFS integration.
https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_...
No reinstall needed ;)
I've personally had drive failures, fs corruptions due to power loss (which is supposed not to happen on a cow filesystem), fs and file corruption due to ram bitflips, etc. All the times btrfs handled the situation perfectly, with the caveat that I needed the help from the btrfs developers. And they were very helpful!
So yeah, btrfs has a bad rep, but it is not as bad as the common feeling makes it look like.
(note that I still run btrfs raid 1, as I did not find real return of experience regarding raid 5 or 6)
ZFS lovers need to stop this CoW against CoW violence.
RAID56 under Btrfs has some caveats but I'm not aware of any annecdata (or perhaps I'm just not searching hard enough) within the past few weeks or months about data loss when those caveats are taken under consideration.
Yeah this is something that makes me consider trying raid56 on it. Though I don't have enough drives to dump my current data while re-making the array :D (perhaps this can be changed on the fly?)
I'll have a look, thanks! I guess failing this will make me test by backup strategy that I have never tested in the past.
For btrfs specifically there is an online calculator [1] that shows you the effective capacity for any arbitrary configuration. I use it whenever I add a drive to check whether it’s actually useful.
1: https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&...
For btrfs specifically there is an online calculator [1] that shows you the effective capacity for any arbitrary configuration. I use it whenever I add a drive to check whether it’s actually useful.
1: https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&...
Here are raw outputs:
$ sudo btrfs fi df /media/storage� Data, RAID1: total=6.34TiB, used=6.22TiB System, RAID1: total=32.00MiB, used=992.00KiB Metadata, RAID1: total=11.00GiB, used=8.71GiB GlobalReserve, single: total=512.00MiB, used=0.00B $ sudo btrfs fi us /media/storage� Overall: Device size: 13.64TiB Device allocated: 12.70TiB Device unallocated: 966.54GiB Device missing: 0.00B Device slack: 0.00B Used: 12.47TiB Free (estimated): 600.31GiB (min: 600.31GiB) Free (statfs, df): 547.55GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no
Data,RAID1: Size:6.34TiB, Used:6.22TiB (98.20%) /dev/sdc 2.53TiB /dev/sdk 2.54TiB /dev/sdf 837.00GiB /dev/sdb 3.40TiB /dev/sdi 3.39TiB
Metadata,RAID1: Size:11.00GiB, Used:8.71GiB (79.22%) /dev/sdc 3.00GiB /dev/sdk 4.00GiB /dev/sdf 2.00GiB /dev/sdb 6.00GiB /dev/sdi 7.00GiB
System,RAID1: Size:32.00MiB, Used:992.00KiB (3.03%) /dev/sdc 32.00MiB /dev/sdk 32.00MiB
Unallocated: /dev/sdc 198.49GiB /dev/sdk 187.49GiB /dev/sdf 92.51GiB /dev/sdb 243.02GiB /dev/sdi 245.02GiB
For btrfs specifically there is an online calculator [1] that shows you the effective capacity for any arbitrary configuration. I use it whenever I add a drive to check whether it’s actually useful.
1: https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&...
Here are raw outputs:
$ sudo btrfs fi df /media/storage� Data, RAID1: total=6.34TiB, used=6.22TiB System, RAID1: total=32.00MiB, used=992.00KiB Metadata, RAID1: total=11.00GiB, used=8.71GiB GlobalReserve, single: total=512.00MiB, used=0.00B $ sudo btrfs fi us /media/storage� Overall: Device size: 13.64TiB Device allocated: 12.70TiB Device unallocated: 966.54GiB Device missing: 0.00B Device slack: 0.00B Used: 12.47TiB Free (estimated): 600.31GiB (min: 600.31GiB) Free (statfs, df): 547.55GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no
Data,RAID1: Size:6.34TiB, Used:6.22TiB (98.20%) /dev/sdc 2.53TiB /dev/sdk 2.54TiB /dev/sdf 837.00GiB /dev/sdb 3.40TiB /dev/sdi 3.39TiB
Metadata,RAID1: Size:11.00GiB, Used:8.71GiB (79.22%) /dev/sdc 3.00GiB /dev/sdk 4.00GiB /dev/sdf 2.00GiB /dev/sdb 6.00GiB /dev/sdi 7.00GiB
System,RAID1: Size:32.00MiB, Used:992.00KiB (3.03%) /dev/sdc 32.00MiB /dev/sdk 32.00MiB
Unallocated: /dev/sdc 198.49GiB /dev/sdk 187.49GiB /dev/sdf 92.51GiB /dev/sdb 243.02GiB /dev/sdi 245.02GiB
they likely have distributed layer on top which takes care of data corruption and losses happening on specific server
Just out of curiosity: is there a specific reason you're not using plain-vanilla filesystems which _are_ stable?
Personal anecdote: i've only ever had serious corruption twice, 20-ish years ago, once with XFS and once with ReiserFS, and have primarily used the extN family of filesystems for most of the past 30 years. A filesystem only has to go corrupt on me once before i stop using it.
Edit to add a caveat: though i find the ideas behind ZFS, btrfs, etc., fascinating, i have no personal need for them so have never used them on personal systems (but did use ZFS on corporate Solaris systems many years ago). ext4 has always served me well, and comes with none of the caveats i regularly read about for any of the more advanced filesystems. Similarly, i've never needed an LVM or any such complexity. As the age-old wisdom goes, "complexity is your enemy," and keeping to simple filesystem setups has always served my personal systems/LAN well. i've also never once seen someone recover from filesystem corruption in a RAID environment by simply swapping out a disk (there's always been much more work involved), so i've never bought into the "RAID is the solution" camp.
I'd guess that it is the classic case of figuring out if something works without using it being a lot harder than giving it a go and seeing what happens. I've accidentally taken out my own home folder in the past with ill-advised setups and it is an educational experience. I wouldn't recommend it professionally, but I can see the joy in using something unusual on a personal system. Keep backups of anything you really can't afford to lose.
And one bad experience isn't enough to get a feel for how reliable something is. It is better to stick with it even if it fails once or twice.
For non-critical subsystems, sure, but certain critical infrastructure has to get it right every time or it's an abject failure (barring interference from random cosmic rays and similar levels of problems). Filesystems have been around for the better part of a century, so should fall into the category of "solved problem" by now. i don't doubt that advanced filesystems are stupendously complex, but i do doubt the _need_ for such complexity beyond the sheer joy of programming one.
> It is better to stick with it even if it fails once or twice.
Like a pacemaker or dialysis machine, one proverbial strike is all i can give a filesystem before i switch implementations.
$ grep bar foo.txt | tr A-Z a-z > foo.txt
If the file isn't in source control, a backup, or auto-synced cloud storage, it can't be _that_ important. If it was in either, it could be recovered easily without replacing one's filesystem with one which needs hand-holding to keep it running. Shrug.
- It can do 4x 15-minute snapshots, 24x hourly snapshots, 7x daily snapshots, 4x weekly snapshots, and 12x monthly snapshots, without making 51 copies of my files.
- Taking a snapshot has imperceptible performance impact.
- Snapshots are taken atomically.
- Snapshots can be booted from, if it's a system that's screwed up and not just one file.
- Snapshots can be accessed without disturbing the FS.
In my experience it hasn't required more hand-holding than ext4 past the initial install, but the OSes that most of my devices use either officially support ZFS or don't use package managers that will blindly upgrade a kernel past what out-of-tree modules I'm using will support, which I think fixes the most common issue people have with ZFS.
my personal reasons are raid + compression
- ZStandard compression is a performance boost on crappy spinning rust
- Snapshots are amazing, and I love being able to quickly send and store them using send and receive
- I like not having to partition the disk at all, and still be able to have multiple datasets that share the same underlying storage. LVM2 has way too many downsides for me to still consider it, like the fact that thin provisioning was quite problematic (i.e. ext4 and the like have no idea they're thin provisioned, ...)
- I like not having to bother with fstab anymore. I have all of my (complex) datasets under multiple boot roots, and I can mount pools from a live with an altroot and immediately get all directories properly mounted
- AFAIK only ZFS and Btrfs support checksums out of the box. I hate the fact that most FS can in fact bitrot and silently corrupt files. With ZFS and Btrfs in theory you can't easily restore your data, but at least you'll know if it got corrupted and restore it from a backup
- I like ZVOL; I appreciate being able to use them as sparse disks for VMs that can be easily mounted without using loopback devices (you get all partitions under /dev/zvol/pool/zvol-partN)
- If you have a lot of RAM,the ZFS ARC can speed up things a lot. ZFS is somewhat slower most of the time than "simpler" FS, but with 10+ GB availble to ARC it's been faster in my experience than any other FS
I do use "classic" filesystems for other applications, like random USB disks and stuff. I just prefer ZFS because the feature set is so good and it's been nothing but stable in day to day use. I've literally had ZERO issues with it in 8+ years - even when using the -git version it's way more stable than Btrfs ever was.
You can do a replace, but then you need to buy a new drive.
Personally, I was one of those people. Very excited about the prospects of btrfs, switched several machines over to it to test, ended up with filesystem corruption and had to revert to ext. Now, whenever I peek at btrfs, I never see anything that's compelling over running ZFS, which I've run for close to 15+ years, and run hard, and have never had data loss. Even in the early days with zfs+fuse, where I could regularly crash the zfs fuse; the zfs+fuse developers quickly addressed every crash I ran into, once I put together a stress test.
Is it really? I must have missed the news. Back when it was released completely raw as a default for many distros, there were fundamental design level issues (e.g. "unbound internal fragmentation" reported by Shishkin). Plus all the reports and personal experiences of getting and trying to recover exotically shaped bricks when volume fills to 100% (which could happen at any time with btrfs). Is it all good now? Where can I read about btrfs behaving robustly when no free space is left?
That's a weird argument. Even if it's true, it is now stable, and has been for a long time. btrfs has long been my default, and I'd be wary of switching to something newer just because someone was mad that development took a long time.