A degraded array should have no substantially greater risk of failure than the single disks the vast majority of people rely on daily. I can think of one obvious reason one might mount the degraded array simply to copy data from it to a different arrangement. Another reason is in a non enterprise use case a desktop users might not instantly have a disk available and might mistakenly believe that this is no more risky than having only one disk in the first place.
This is ridiculously flaky for something that is supposed to be years in development with backing from major companies.
This confirms my feeling that btrfs belongs in the waste basket.
Don't nitpick about what a two-word summary implies when there's only one underlying bug at issue. Just address the bug itself. "Production ready" never means 100% bug-free.
> A degraded array should have no substantially greater risk of failure than the single disks the vast majority of people rely on daily.
If you are comfortable dropping from the RAID1 profile to a non-RAID profile, btrfs requires you to explicitly make this conversion; there's no safe way to make it automatic. Forcing the filesystem to accept writes while it's still in RAID1 mode but is incapable of providing RAID1 data integrity is something that you should expect to cause problems.
> I can think of one obvious reason one might mount the degraded array simply to copy data from it to a different arrangement.
You can mount read-only as many times as you want if you're going to copy the remaining data to a different filesystem. If you want to do an in-place recovery, the current limitation is only that you shouldn't mount the degraded array as writable until you're ready to make it not degraded anymore.
This might make sense for you, but is insane as a default policy. Manual intervention should be required before the FS will accept writes with less than the configured degree of redundancy. Silently mounting and hoping the user notices something in their logs is too dangerous.
> I created a raid1 btrfs filesystem by converting an existing single btrfs instance into a degraded raid1, then added the other drive
This seems backwards. Why not add the second drive, then convert to RAID1?
-
Note that the patch to work around the refusal to mount is extremely simple, and that the patch is quite safe if used properly. But it's not really an acceptable solution for upstreaming, because it will lead to bigger problems if used in more complicated situations. There are several potential solutions that would be safe and widely deployable, but all involve changing far more than two lines of code.
For example in a large cluster where that degradation is expected and monitored for.
How do you add a second drive? Is there a brtfs raid faq/howto?
In environments with a smaller budget that want to get the system back up and running before a replacement drive is available, it could be valuable to be able to pre-specify that the system should rebalance with less redundancy when a drive goes missing. I'm not aware of any work to implement this kind of feature. No enterprise customer would want or use this feature, and even a home user on a shoestring budget wouldn't necessarily want this rebalancing to happen automatically. (What if the drive was only temporarily missing, such as from a failed or loose SATA cable? You wouldn't want to do a ton of re-writing of data only to have to reverse it on the next boot.)
That's what RAID1 usually means.
And sure, you can't survive the next without replacing the failed drive and resyncing.
But the remount read only thing is something different. It's a useful failure mode, but doesn't help with operational simplicity.
(If the SATA cable is loose, the it'll cause intermittent failures, you'll see it in the log, and there will be a lot of resync events. And probably degraded performance, a lot of ATA (or SCSI errors in case of SAS), and other bus/command errors that go away on retry. And with SMART it's possible to at least guess that it's not the drive. It'd be great to have an error notification interface from the kernel and a tool could try to dig into the relevant subsystem's perf and health data to try to guess what's the faulty component exactly.)
The way to encounter it is to mount the array a second time. That's hardly "cavalier handling".
A decade in, and RAID 1 doesn't work right. Saying this clearly offends the feels here at HN, but it's a fact.
`mount` is not the same as `mount -o degraded,rw`. The latter should raise the eyebrows of anyone paying attention to what they're typing. Odds are that you'll have to consult the docs to even find these mount options, because it doesn't happen automatically. This is where any careful, sane user who's concerned about their data would spend a few more minutes thinking through their entire recovery procedure.
There's one corner case to RAID1 recovery where the current tooling does not fully prevent a careless user from putting the FS in a bad state. This is not the same as "RAID1 doesn't work right".
What is avoidable is that btrfs could make it harder to do the equivalent of `mount -o degraded,rw`. But ultimately, there will be some mechanism for modifying a degraded array, and it'll get documented and then excerpted in blog posts and StackOverflow answers without all the context, and users will find a way to work themselves into a corner. There are all kinds of ways to do this with ZFS, too. ZFS tends to default to the approach of requiring you to copy all your data elsewhere and rebuild your array from scratch. What btrfs is doing here is no worse, except that it's a bit less up-front about the limitation because it's actually completely avoidable and this is a fixable UI bug, not a deep-seated architectural limitation.
Basically, btrfs doesn't want to allow a writeable mount when it might be missing some data. If there's some data on the FS that isn't stored with the RAID1 profile, then the kernel can't safely assume that the missing drive didn't have more chunks like that, holding data that wasn't mirrored on one of the surviving drives. But it's currently not possible to convert from RAID1 to non-RAID or to rebuild the array with a replacement without mounting the degraded array as writeable, which leads to non-RAID data being written. That puts the FS in a state that cannot be automatically judged safe at mount time, and the FS remains in that state until the recovery is complete (either converting from RAID1 to non-RAID, or replacing the failed drive).
There's no easy way to require the user to specify at the time of the `mount -o degraded,rw` whether they intend to resolve the situation by ceasing to use RAID1 or by replacing the failed drive. That leaves users with the opportunity to do neither and instead make the situation worse.
At least for RAID1, it seems that implementing RAID1 N-way mirroring would ease the process to recover from a failed drive. In case of drive failure, we could use the remaining drive in read-only mode to copy the data to a new drive, hence creating a RAID1 array with two working drives and one failed drive. The OS should then allow to boot in rw mode, and from there it is easy to remove the failed drive from the RAID1 array.
However it seems that RAID1 N-way mirroring (with N > 2) is not even on the roadmap at this moment.
Have I misunderstood something or does this approach make sense ?
Being able to do N-way mirroring with three or more copies of the data would be nice, but it's not necessary; 2-way mirroring across 3 or more drives is sufficient, and the hot spare feature will be more widely useful.
I was referring to this sequence of events: 1) 2-way mirroring across 2 drives 2) one drive fails 3) buy and plug a new drive 4) rebalance to have 3-way mirroring across 3 drives (with one being out): this is currently not possible 5) remove the failed drive, ending with 2-way mirroring across 2 drives
But it seems that you are referring to: 1) 2-way mirroring across 3 drives 2) one drive fails 3) rebalance to have 2-way mirroring across the 2 working drives 4) remove the failed drive, ending with 2-way mirroring across 2 drives
I assume that people don't/won't start the initial RAID1 with 3 drives.
Anyway, I would find 3-way mirroring across 3 drives very useful as it gives a simple identical foolproof process to replace a faulty hard drive, whether it has just a few corrupted data (but still readable) or have completely failed : just plug a new drive, rebalance, reboot and remove the defective drive.
I'm not sure this even has meaning. But anyways, it's probably pointless to try to kick off a rebalance when the FS is still trying to use a dead drive. Either use the device replace command (which isn't stable yet), or tell btrfs to delete the dead drive then add the replacement drive. If the problem drive is failing but not completely dead yet, then the device replace command is supposed to move data over with a minimum of excess changes to drives other than the ones being removed and added. But the device replace command doesn't properly handle drives with bad sectors yet, so the separate remove and add actions are more reliable albeit slower and put more work on the other drives in the array.
Where can I find the proper official procedure to replace a failed drive (i.e. which cannot be mounted anymore) in a RAID1 array (with 2 drives) ?
I found these 2 links: https://unix.stackexchange.com/questions/334228/btrfs-raid1-... https://unix.stackexchange.com/questions/227560/how-to-repla...
But it is written there that if 'replace' doesn't work, it can take up to 5 days using add/remove for only 100GB !
I haven't found any official procedure on the Btrfs wiki.
That doesn't seem to match anything I've seen. The btrfs status page classifies RAID1 and RAID10 features as "mostly OK". The one and only documented caveat is that when a disk in a RAID1 fails, you should mount the filesystem as read-only until you are ready to fix the problem by either replacing the failed disk or converting to a non-RAID profile. There's a real bug underlying this limitation, but the only way to encounter this bug is to be quite cavalier about how you handle a degraded array, by doing something that you shouldn't expect to be safe.