News for hackers. Stuff that matters.
Re:Bringing Bcachefs to Linux Mainline by winter_blue on Wednesday May 18, 2022 @07:53AM (#31421638)
> A lot of people that got burned with Btrfs

How did they get burned? (I'm thinking of switching to Btrfs on my personal laptop (running Arch), mainly for the snapshots, subvolumes, checksums+deduplication, etc.)

Re:Bringing Bcachefs to Linux Mainline by yjftsjthsd-h on Wednesday May 18, 2022 @09:58AM (#31422844)
I lost a root filesystem twice, on OpenSUSE in a one-disk system (i.e. what should have been the most stable well tested configuration), around 2018-19 IIRC. It now appears most likely that the system ran out of free space (suse ships snapper set to snapshot on every package operation, and the partition was smallish), and that apparently triggered corruption that ... at least, I couldn't figure out how to fix. Power loss may have been involved. Details fade from my memory, but the violent aversion to ever trusting it remains.
Re:Bringing Bcachefs to Linux Mainline by chasil on Wednesday May 18, 2022 @01:33PM (#31425417)
It is actually wise to shrink a btrfs filesystem on its partition by a few megabytes, so some expansion room is available in an emergency.

There are very few filesystems that can shrink like this:

  btrfs filesystem resize -1G /mount
If the "btrfs fi df" command shows any of the data/system/metadata elements close to limits, it is time to take action. They can all push you into read-only mode.
Re:Bringing Bcachefs to Linux Mainline by zepearl on Wednesday May 18, 2022 @02:44PM (#31426221)
I lost a btrfs-raid5-like NAS out of the blue (after 3-4 months of using it, I didn't even write a lot on it):

the filesystem was always cleanly unmounted, I didn't do any crazy experiments with it, it had plenty (>40%, hundreds of GiBs) of free disk space; on one day I noticed that accessing some files was slower than usual, the noise of the HDDs had a weird pattern and from there it went downhill (at that time there was no fsck utility so I didn't even know what to do - if I'm not mistaken, BUT I MIGHT, the general opinion was that a fsck-util was not even needed? but again, I might have just dreamed this) and finally stopped working with a huge amount of error messages in the kernel logs => I had to reformat everything (I went back to mdadm-raid5) and restore from the secondary NAS' backup.

In my opinion there were 2 main things that contributed to the bad reputation of the FS:

1)

In those days there was a looot of hype about btrfs (as well for me it was a dream come true) which is the reason why I decided to use it.

As far as I remember I didn't have to enable any special "experimental" feature in the kernel to use it => that gave me the impression that "it works under normal conditions, the DEVs are probably being a bit conservative for corner cases".

2)

All functionalities (e.g. raid5 with rebalancing in my case) were all available from day 1 (at least when I started using it) so if something's available I tend to think that it works (I mean, from a user's point of view why make something available that does not work?) - but no, apparently many people had many problems with many different configurations/usage patterns.

-

As others have written I hope that Bcachefs will primarily focus on reliability and only secondarily on performance & features, but I admit that that's difficult to balance :). And I hope that the fsck-utility will be available from day1 (at least with a reduced functionality). And I hope that "experimental" features will need some kind of very explicit command line option (e.g. "-might_break_everything") to use them through userland utils.

Go Kent!!! :) I'm since a while one of your Patreon follower - If you manage to create something usable I'll print the pic shown on LWM and I'll hang it up in my PC-room, maybe I'll even take my twitter account out of cryosleep to tell Musk that he should use Bcachefs as FS for his Teslas :))

Re:Bringing Bcachefs to Linux Mainline by sph on Thursday May 19, 2022 @05:03AM (#31432302)
RTFM

"The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%."

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...

RAID56 Stability: Unstable (do not use for other then testing purposes, known severe problems, missing implementation of some core parts)

https://btrfs.wiki.kernel.org/index.php/Status

Re:Bringing Bcachefs to Linux Mainline by chasil on Wednesday May 18, 2022 @01:07PM (#31425145)
Many people used btrfs while it was still experimental, and lost data.

Attempting to create a btrfs filesystem returned a prompt with this warning (until 2013):

  Btrfs is a new filesystem with extents, writable snapshotting,
  support for multiple devices and many more features.

  Btrfs is highly experimental, and THE DISK FORMAT IS NOT YET
  FINALIZED. You should say N here unless you are interested in
  testing Btrfs with non-critical data.
That prompt was removed when things stabilized, and Fedora's use of btrfs as the default root has not been accompanied by complaints of excessive filesystem corruption (although there are still many rough edges and unfinished features).

https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...

Re:Bringing Bcachefs to Linux Mainline by techdragon on Wednesday May 18, 2022 @08:01AM (#31421687)
Since Btrfs was arguably available in the kernel too soon. Some people who should have avoided a new filesystem for lots of reasons ranging from missing features that were in the plan but not implemented yet or in the earliest days not yet stable... many of these early experimenters developed and fostered a vocal belief that Btrfs was "bad" (there's too much to sum up honestly, you could write an entire decent length blog post on the history of Btrfs, its reception and its criticism) and this group has taken a long time to accept that Btrfs may be ok now. Theres still people with the perception that its not as good or ready enough and should not be used.
Re:Bringing Bcachefs to Linux Mainline by blueflow on Wednesday May 18, 2022 @08:21AM (#31421848)
From the bcachefs homepage:

> We prioritize robustness and reliability over features and hype: we make every effort to ensure you won't lose data.

This is exactly what btrfs failed at, this is not just an issue with the code itself, but also with the developers. Why should i trust feature-oriented developers with writing my infrastructure code, where reliability and robustness are top priority?

Re:Bringing Bcachefs to Linux Mainline by viraptor on Wednesday May 18, 2022 @08:23AM (#31421875)
All this, but also two specific cases:

There's a list of features and readiness at https://btrfs.wiki.kernel.org/index.php/Status - there were people who didn't check it and used unstable features and experienced issues. This is partially on the project for not making it more explicit.

The second is that there are people who read that, see the unstable raid5/6, and are vocal about btrfs being overall broken / not ready rather than accepting that feature just not being available yet.

Re:Bringing Bcachefs to Linux Mainline by zozbot234 on Wednesday May 18, 2022 @08:26AM (#31421902)
A file system that can eat your data is broken. If a feature is not ready for public consumption, it should not be built-in in the first place.
Re:Bringing Bcachefs to Linux Mainline by viraptor on Wednesday May 18, 2022 @08:31AM (#31421950)
Then every filesystem is broken by that definition. You're setting an impossible standard given how non-specialist software is developed.

Here's a list of failures from zfs, including many kernel panics for example https://github.com/openzfs/zfs/labels/Type%3A%20Defect - is it broken and shouldn't have been released?

Re:Bringing Bcachefs to Linux Mainline by Beldin on Wednesday May 18, 2022 @10:13AM (#31423034)
Fine, let's be more nuanced: no filesystem should lose data if FAT16/FAT32/NTFS 3.1/EXT2/EXT3 wouldn't.
Re:Bringing Bcachefs to Linux Mainline by vlowther on Wednesday May 18, 2022 @10:34AM (#31423284)
well, FAT16/FAT32/EXT2 are not journalled, so making them lose data in a giant variety of ways is a trivial affair. You might want a higher bar than that.
Re:Bringing Bcachefs to Linux Mainline by Beldin on Wednesday May 18, 2022 @05:56PM (#31428150)
That's the point: if file systems that basic would NOT lose data, then it is unacceptable if another file system does.

Maybe to put it differently: new file systems should offer improvements wrt reliability for stable features of the file systems of the previous generation; regressions are unacceptable.

Re:Bringing Bcachefs to Linux Mainline by Spivak on Wednesday May 18, 2022 @10:56AM (#31423582)
Yeah, when you get down to it the opinion really boils down to "the file system is broken iff it doesn't lose data for the use-cases I personally use it for."
Re:Bringing Bcachefs to Linux Mainline by orra on Wednesday May 18, 2022 @09:11AM (#31422332)
> that feature just not being available yet.

How long are we expected to wait? It's been over a decade.

btrfs was touted as a spiritual successor to ZFS. raidz5 is a core feature of ZFS, not an incidental extra feature.

Re:Bringing Bcachefs to Linux Mainline by matheusmoreira on Wednesday May 18, 2022 @12:50PM (#31424939)
We've been waiting so long for proper software RAID support that ZFS got RAID-Z expansion before it happened. I no longer have any reason to use btrfs.
Re:Bringing Bcachefs to Linux Mainline by chasil on Wednesday May 18, 2022 @01:36PM (#31425462)
The one thing that btrfs can do that ZFS cannot is defragment. Crippling a ZFS dataset on rotational media does not appear to be at all difficult.

https://www.usenix.org/system/files/login/articles/login_sum...

Re:Bringing Bcachefs to Linux Mainline by alexghr on Wednesday May 18, 2022 @11:25AM (#31423954)
I'm going to add my N=1 anecdata if no one minds:

I've been running full Btrfs on my workstation since December 2021 with 0 issues (kernel 5.10). It survived multiple hard reboots and power offs, Ryzen 5000 CPUs seems to lose power every now and then on Linux (it's a known issue for years now that no one has figured out yet).

Having compression is great. compsize reports I've saved 33GB on my /home subvolume so far (https://github.com/kilobyte/compsize). I don't use snapshots, I just backup the whole thing.

Obviously this doesn't mean that in the future Btrfs won't eat my data, but so far it hasn't and I'm happy.

Re:Bringing Bcachefs to Linux Mainline by pmarreck on Wednesday May 18, 2022 @10:40AM (#31423357)
I lost not one but TWO entire volumes to it in the past few years. Just one day they wouldn't boot. I wasn't using encryption and I wasn't using any RAID feature, just compression. The people who think BTRFS is a viable substitute for ZFS have just gotten lucky with not getting burned.

There's also the issue with BTRFS performance degrading hugely with any significant fragmentation, to the point that "autodefrag" has to be a feature, which of course also adds SSD wear...

ZFS has never burned me.

Re:Bringing Bcachefs to Linux Mainline by oxplot on Wednesday May 18, 2022 @08:31AM (#31421953)
> How did they get burned?

In the early days after Fedora made it available, I lost my root twice after a kernel panic and later after a hard power reset.

EDIT: in comparison, I've been using xfs for almost a decade now with numerous hard power offs and kernel panics. I've only ever lost select files that were open for write at the time of restart, and not the entire filesystem.

Re:Bringing Bcachefs to Linux Mainline by mekster on Wednesday May 18, 2022 @10:18AM (#31423089)
I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago.

Never used btrfs but I see corruption reports in here everytime I read about it, and it's been in development for over a decade and this thing needs to be put to rest before it eats more people's data.

Hopefully bcachefs becomes feature parity and stability against zfs to provide a modern filesystem to wider Linux audience at last.

Funny to see we're still using ext4 as the default filesystem which is nothing fancy forever.

Re:Bringing Bcachefs to Linux Mainline by dralley on Wednesday May 18, 2022 @10:59AM (#31423627)
> I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago.

Because the Fedora community wanted to, basically. They're not obligated to go along with whatever Red Hat wants to do, although there is naturally some alignment there. Plus having a different default filesystem really isn't that big of a deal.

disclosure: I work for Red Hat.

Re:Bringing Bcachefs to Linux Mainline by mekster on Wednesday May 18, 2022 @11:09AM (#31423759)
Well, I always hear about data loss incidents whenever btrfs shows up in HN comments, so I thought making it a default in Fedora desktop sounded like a big deal as there must've been a good reason on why RH dropped it.
Re:Bringing Bcachefs to Linux Mainline by sph on Wednesday May 18, 2022 @04:34PM (#31427283)
Maybe it's always the same stories. I haven't read of any happening in the past couple years. Of the two I've read in this thread, one happened ages ago, the other involves raid5 which AFAIK is not recommended for production use. Also, kernel bugs tend to get fixed, especially now that btrfs is recommended by a few distros.
Re:Bringing Bcachefs to Linux Mainline by dralley on Wednesday May 18, 2022 @11:20AM (#31423898)
I know nothing about the reasons it was dropped (apart from what I've read on HN, ironically) but keep in mind this was years ago, when it was still deeply, deeply troubled and there was a lot of churn going on. "Churn" and "stable" kernels like RHEL don't mix well.
Re:Bringing Bcachefs to Linux Mainline by warmwaffles on Wednesday May 18, 2022 @09:00AM (#31422207)
I've been running in danger mode of RAID6 on a 15 drive NAS here at my home for the better part of 5 years now, and I have had zero issues. The server is also running Arch and I run package updates every two weeks or so. Maybe I am just lucky.
Re:Bringing Bcachefs to Linux Mainline by Mister_Snuggles on Wednesday May 18, 2022 @10:50AM (#31423500)
I've been burned enough times to avoid it.

My Linux laptop, running OpenSuse Tumbleweed, got its root fs into a state where it would hang when trying to mount RW. I was able to boot off of a rescue image, mount it RO, and copy my data to another machine. None of the repair suggestions were able to get the fs into a mountable state, I had to reinstall it and copy my data back.

Every VM at work that uses btrfs gets into a state where some maintenance process (rebalancing, I think - I'm not one of the Linux admins) will consume all CPU in the VM and prevent it from doing any work. The only thing that varies is how long it takes - it seems to be related to how much write activity there is on the fs, VMs with heavier write activity get into this state faster. This has led to outages and the Linux vendor was unable to resolve this issue. We've since removed all traces of btrfs from our data center.

Re:Bringing Bcachefs to Linux Mainline by cyphar on Wednesday May 18, 2022 @11:01AM (#31423648)
If you used Docker on your machine, this was a bug in the remount code (they forgot to re-trigger the qgroup scan if it was interrupted when you did an rw-remount) which I fixed several years ago[1]. I don't know what that will do for your confidence in btrfs, but hopefully it'll give you some closure.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re:Bringing Bcachefs to Linux Mainline by Mister_Snuggles on Thursday May 19, 2022 @09:14AM (#31434202)
The incident with my Linux laptop was definitely not Docker-related, but the description of the issue sounds like it could still apply.

We didn't use Docker at work, at least not on the VMs that I was affected by. This one was a gradual degradation over time - the maintenance process would run fine, but after a while it would start hanging the machine. It really felt like the fs was only good for some amount of writes before the maintenance process got overwhelmed and would hang the VM whenever it ran.

I'm happy to see that things are getting better, but the reputational damage will take a long time to repair.

All that being said, I still use btrfs on my Linux laptop, but I've also become more conscientious about backing it up.

Re:Bringing Bcachefs to Linux Mainline by cyphar on Thursday May 19, 2022 @10:22AM (#31435025)
I was told a while ago that rebalancing could get into pathological cases fairly easily (on openSUSE we used to run a rebalance in a cron job and it turns out this would grind machines to a halt). I still use btrfs and haven't had this issue for several years so maybe the issue has been resolved? I'm not sure what the resolution might have been.
Re:Bringing Bcachefs to Linux Mainline by rkeene2 on Wednesday May 18, 2022 @01:07PM (#31425144)
BtrFS is the only filesystem I've ever lost data with, and I've used a lot of filesystems over the decades (UFS, VxFS, JFS, XFS, EXT2/3/4, ZFS).

I have a comparison of features between BtrFS, ZFS, and EXT4+LVM at [0], and as I note at the top of that post I was looking to move to BtrFS. I did so and lost data.

[0] https://rkeene.org/projects/info/wiki/BtrFS

Re:Bringing Bcachefs to Linux Mainline by rodgerd on Wednesday May 18, 2022 @08:53AM (#31422129)
The main thing I found btrfs was good for was testing the restore process of my backups. I was using it in supposedly known-good configurations (RAID 1 or RAID 10) and it consistently destroyed data.

The fact that it turned out that the scrub function spent years corrupting data by mistake was only one of the many flaws with it.

Re:Bringing Bcachefs to Linux Mainline by EspadaV9 on Saturday May 21, 2022 @05:19AM (#31456468)
I'd recommend listening to some of the 2.5 Admins podcasts. There are a few episodes (earlier on I think) where one of the hosts lists a number of issues with how it is implemented and basically wouldn't recommend it for anything, even in trivial set ups with a single drive there are far better alternative's.