ZFS is extremely annoying with the way it does extend and the fact that you can’t mismatch drive size. It’s not a panacea. There clearly is space for an improved design.
It clearly is an acceptable one for a lot of people but it does leave space for alternative designs.
[1]: https://hexos.com/blog/introducing-zfs-anyraid-sponsored-by-...
Over the last one or two years I've experienced twice a checksum mismatch on the file storing the memory of a VMWare Workstation virtual machine.
Both are very likely bugs in Btrfs, and it's very unlikely that have been caused by the user (me).
In the relatively far past (around 5 years ago), I've had the system (root being on Btrfs) turning unbootable for no obvious reason, a couple of times.
The question was around usage, because without knowing people's usecases and configurations it'll never be usable for you while working fine for others.
As usual with all these Linux debates, there's a loud group grinding their old hatreds that can be decade old.
If 0.1% of users say it corrupted for them, and then don't provide any further details and no one can replicate their scenario then it does make it hard to resolve it
there's a feedback effect: if users know that a filesystem takes these kinds of issues seriously and will drop what they're doing and jump on them, a lot of users will very happily spend the time reporting bugs and working with devs to get it resolved.
people don't like wasting their time on bug reports that go into the void. they do like contributing their time when they know it's going to get their issue fixed and make things better for everyone.
this is why I regularly tell people "FEED ME YOUR BUG REPORTS! I WANT THEM ALL!"
it's just what you have to do if you want your code to be truly bulletproof.
No, Kent, they are not. Posting attacks like this without evidence is cowardly and dishonest. I'm not going to tolerate these screeds from you about people I've worked with and respect without calling you out.
Every time you spew this toxicity, a chunk of bcachefs users reformat and walk away. Very soon, you'll have none left.
It doesn't matter if you want to "tolerate" it if every time there's a filesystem thread there's stories about lost filesystem and unfixed bugs, and the rare times people try to report bugs or issues they go nowhere.
You're ranting about reality, and somehow I doubt you were ever a user of mine.
I've worked hard to establish an engineering community where no one has to be afraid to point out broken shit, including and especially in my code, and I wouldn't want your attitude anywhere near it.
Absolutely. Demonstrably much moreso than you over the past decade.
> I've worked hard to establish an engineering community where no one has to be afraid to point out broken shit,
You have failed. Look at this conversation: https://lore.kernel.org/lkml/fe51512baa18e1480ce997fc535813c...
Do you think Arnd, David, and Russell feel like they were rewarded for pointing out your broken shit? I doubt it. I certainly don't. You were so completely and obviously in the wrong there I have difficulty believing it was real.
I'm not replying to you again. Good luck :)
Not comparable to losing filesystems, sorry...
Also, filesystems just work. Nobody is gonna say "oh I'm using fileystem X and it works!" because that's the default. So, naturally, 99% of the stuff you'll hear about filesystems is when they don't work.
Don't believe me? Look up NTFS and read through reddit or stackexchange or whatever. Not a lot of happy campers.
I don't.
It's also the reason I am completely against the Debian "backporting" methodology to pretend they aren't using the new version of something.
Debian doesn't "backport" anything, they ship upstream LTS kernels straight, so please stop spreading this misinformation.
For some bizarre reason I can't conceive people not only believe but also keep complaining that Debian ships franken-kernels like RHEL does. See [0] for another instance.
Is it wrong to ask how to reproduce an issue?
I then had to manually delete the file with the I/O error (for which I had to resovle the inode number it barfed into dmesg) and try again - until the next I/O error.
(I'm still not sure if the disk was really failing. I did a full wipe afterwards and a full read to /dev/null and experienced no errors - might have just been the meta-data that was messed up)
Despite not being directly used, these blocks are kept (and cannot be reused) because another part of the extent they belong to is actually used by files.
This can happen if a large file is written in one go, and then later one block is overwritten - btrfs may keep the old extent which still contains the old copy of the overwritten block.It's generally fine if you stay on the happy path. It will work for 99% of people. But if you fall off that happy path, bad things might happen and nobody is surprised. In my personal experience, nobody associated with the project seems to trust a btrfs filesystem that fell off the happy path, and they strongly recommend you delete it and start from scratch. I was horrified to discover that they don't trust fsck to actually fix a btrfs filesystem into a canonical state.
BCacheFS had the massive advantage that it knew it was experimental and embraced it. It took measures to keep data integrity despite the chaos, generally seems to be a better design and has a more trustworthy fsck.
It's not that I'd trust BCacheFS, it's still not quite there (even ignoring project management issues). But my trust for Btrfs is just so much lower.
Btrfs has destroyed itself on my testing/lab machines three times during last two years up to point where recovery wasn’t possible. Metadata corruption being main issue (or that’s how it looks like to me at least).
As of now I trust BCacheFS way more. I’ve given it roughly the same time to prove itself as Btrfs too. BCacheFS has issues but so far I’ve managed to resolve them without major data loss.
Please note that I currently use ext4 in all ”really important” desktop/laptop installations and OpenZFS in my server. Performance being the main concern for desktop and reliability for server.
source: worked as a support engineer for a block storage company, witnessed hundreds of customers blowing one or both of their feet off with ZFS.
(Legitimate question: I manage several PB with ZFS and would like to know where I should be more cautious.)
There were legitimate bugs in ZFS that we hit. Mostly around ZIL/SLOG and L2ARC and the umpteen million knobs that one can tweak.
You can do the same with just about any file system. In the Windows world you can blow your feet off with NTFS configuration too.
Of course there have been bugs, but every filesystem has had data-impacting bugs. Redundancy and backups are a critical caveat for all file systems for a reason. I once heard it said that “you can always afford to lose the data you don’t have backed up”. I do not think that broadly applies (such as with individuals), but it certainly applies in most business contexts.
Obviously there's footguns in everything. Filesystem ones are just especially impactful.
But a lot of people out there will slap a bunch of USB 2.0 hard drives on top of an old gaming computer. I’m all for experimenting, and I sympathize that it’s expensive to run ZFS on “ZFS class” platforms and hardware. I don’t begrudge others that.
It would be really nice if there was something like ZFS that was a tad more flexibility and right in the kernel with consistent and concise user space tooling. Not everyone is comfortable with DKMS.
Can you provide some specifics? So far all I see is vague complains with no substance, and when complainers are lightly pressed they go defensive.
I did live specific situations over several years in a support engineer role where a double digit percentage of customers in enterprise configurations that ended up somewhere between terrible performance and catastrophic data loss due to the misunderstood configuration of a very complex piece of software.
If you wanna use ZFS, use ZFS. I'm not the internets crusader against it. I have no doubt there's thousands of PB out there of perfectly happy, well configured and healthy zpools. It has some truely next-gen features that are extremely useful. I've just seen it recommended so, so many times as a panacea when something simpler would be just as safe and long lasting.
It's kinda like using Kubernetes to run a few containers. Right?
I see.
> I did live specific situations over several years in a support engineer role where a double digit percentage of customers in enterprise configurations that ended up somewhere between terrible performance and catastrophic data loss due to the misunderstood configuration of a very complex piece of software.
I'm sorry, but this claim is outright unbelievable. If the project was even half as unstable as you claim to be, no one would ever use it in production at all. Either you are leaving out critical details such as non-standard patches and usages that have no relationship with real world usage, or you are fabricating tales.
Also, it's telling that no one stepped forward to offer any concrete details and specifics on these hypothetical issues. Makes you think.
I'm not being melodramatic. You can take a story or leave it. I'm not here to convince you one way or another.
And frankly I don't particularly appreciate being called a liar, btw. Nice use of quoting also. Good day.
There are workarounds, with their respective caveats and warnings.
The phrasing of this tends me to believe that the customers set up ZFS in a 'strange' (?) way. Or was this a bug(s) with-in ZFS itself?
Because when people talk about Btrfs issues, they are talking about the code itself and bugs that cause volumes to go AWOL and such.
(All file systems have foot-guns.)
There was a _very_ nasty bug in the ZFS L2ARC that took out a few PB at a couple of large installations. This was back in 2012/2013 when multiple PBs was very expensive. Was a case of ZFS putting data from the ARC into the pool after the ZIL/SLOG had been flushed.
I don't plan on giving ZFS or other filesystems not designed for Linux another go.
Heck no native filesystem besides btrfs has compression, I'm saving HUNDREDS of GB with zstd compression on my machines with basically zero overhead
bcachefs isn't going away.
The SuSE guy also reversed himself after I asked; Debian too, so we have time to get the DKMS packages out.
As I predicted, out of tree bcachefs is basically dead on arrival - everybody interested is already on ZFS, btrfs is still around only because ZFS can't be mainlined basically