News for hackers. Stuff that matters.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by hinkley on Saturday August 19, 2023 @03:45PM (#37192381)
I’m frustrated because this feature was mentioned by Schwartz when it was still in beta. I thought a new era of home computing was about to start. It didn’t, and instead we got The Cloud, which feels like decentralization but is in fact massive centralization (organizational, rather than geographical).

Some of us think people should be hosting stuff from home, accessible from their mobile devices. But the first and to me one of the biggest hurdles is managing storage. And that requires a storage appliance that is simpler than using a laptop, not requiring the skills of an IT professional.

Drobo tried to make a storage appliance, but once you got to the fine print it had the same set of problems that ZFS still does.

All professional storage solutions are built on an assumption of symmetry of hardware. I have n identical (except not the same batch?) drives which I will smear files out across.

Consumers will never have drive symmetry. That’s a huge expenditure that few can justify, or much afford. My Synology didn’t like most of my old drives so by the time I had a working array I’d spent practically a laptop on it. For a weirdly shaped computer I couldn’t actually use directly. I’m a developer, I can afford it. None of my friends can. Mom definitely can’t.

A consumer solution needs to assume drive asymmetry. That day it is first plugged in, it will contain a couple new drives, and every hard drive the consumer can scrounge up from junk drawers - save two: their current backup drive and an extra copy. Once the array (with one open slot) is built and verified, then one of the backups can go into the array for additional space and speed.

From then on, the owner will likely buy one or two new drives every year, at whatever price point they’re willing to pay, and swap out the smallest or slowest drive in the array. Meaning the array will always contain 2-3 different generation of hard drives. Never the same speed and never the same capacity. And they expect that if a rebuild fails, some of their data will still be retrievable. Without a professional data recovery company.

which rules out all RAID levels except 0, which is nuts. An algorithm that can handle this scenario is consistent hashing. Weighted consistent hashing can handle disparate resources, by assigning more buckets to faster or larger machines. And it can grow and shrink (in a drive array, the two are sequential or simultaneous).

Small and old businesses begin to resemble consumer purchasing patterns. They can’t afford a shiny new array all at once. It’s scrounging and piecemeal. So this isn’t strictly about chasing consumers.

I thought ZFS was on a similar path, but the delays in sprouting these features make me wonder.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by mrighele on Saturday August 19, 2023 @04:12PM (#37192673)
> I’m a developer, I can afford it. None of my friends can. Mom definitely can’t.

The only thing that can work for your mom and your friend is, in my opinion, a pair of disks in mirror. When the space finished, buy another box with two other disk in mirror. Anything more than this is not only too complex for the average user but also too expensive.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by hinkley on Saturday August 19, 2023 @04:32PM (#37192864)
Keeping track of a heterogenous drive array is just as big an imposition.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by Quekid5 on Saturday August 19, 2023 @04:36PM (#37192901)
I think I would advise against a direct mirroring -- instead, I'd do sync-every-24-hours or something similar.

Both schemes are vulnerable to the (admittedly rarer) errors where both drives fail simultaneously (e.g. mobo fried them) or are just ... destroyed by a fire or whatever.

A periodic sync (while harder to set up) will occasionally save you from the deleting the wrong files which mirroring doesn't.

Either way: Any truly important data (family photos/videos, etc.) needs to be saved periodically to remote storage. There's no getting around that if you really care about the data.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by kalleboo on Saturday August 19, 2023 @09:36PM (#37195168)
> A periodic sync (while harder to set up) will occasionally save you from the deleting the wrong files which mirroring doesn't

Which ever solution for disks you end up with, you should definitely always be using something that keeps around periodic snapshots.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by Quekid5 on Monday August 21, 2023 @05:40PM (#37215705)
Agreed, but even that is vulnerable to software bugs around snapshotting itself.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by bartvk on Saturday August 19, 2023 @05:51PM (#37193663)
Even that sounds complex.

If they want network attached storage, I’d just use a single disk NAS, and remotely back it up.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by Osiris on Saturday August 19, 2023 @04:03PM (#37192567)
I completely agree. To build my array I had to buy several drives at the same time. To expand I had to buy a new drive, move the data onto the array, and then I’m left with the extra drive I had too buy to temporarily store the data because I can’t add it to the array.

I would love to have more options for expandable redundancy.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by ilyt on Saturday August 19, 2023 @09:08PM (#37195002)
I just...leave it then create new RAID out of free space once I get more than 2-3 disks that are bigger. But yeah, PITA.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by gregmac on Saturday August 19, 2023 @04:55PM (#37193094)
I can afford it, but have a hard time justifying the costs, not to mention scrapped (working) hardware and inconvenience (of swapping to a whole new array).

I started using snapraid [1] several years ago, after finding zfs couldn't expand. Often when I went to add space the "sweet spot" disk size (best $/TB) was 2-3x the size of the previous biggest disk I ran. This was very economical compared to replacing the whole array every couple years.

It works by having "data" and "parity" drives. Data drives are totally normal filesystems, and joined with unionfs. In fact you can mount them independently and access whatever files are on it. Parity drives are just a big file that snapraid updates nightly.

The big downside is it's not realtime redundant: you can lose a day's worth of data from a (data) drive failure. For my use case this is acceptable.

A huge upside is rebuilds are fairly painless. Rebuilding a parity drive has zero downtime, just degraded performance. Rebuilding a data drive leaves it offline, but the rest work fine (I think the individual files are actually accessible as they're restored though). In the worst case you can mount each data drive independently on any system and recover its contents.

I've been running the "same" array for a decade, but at this point every disk has been swapped out at least once (for a larger one), and it's been in at least two different host systems.

[1] https://www.snapraid.it/

Re:OpenZFS – Incremental add of disks to existing RAIDZ by RulerOf on Saturday August 19, 2023 @11:09PM (#37195706)
I gave snapraid a serious look a few months back and decided it might not be for me because the act of balancing writes out to the "array" member disks appeared to be manual.

I didn't want to point applications at 100 T of "free" space only for attires to start blocking after 8.

Am I mistaken about that?

Re:OpenZFS – Incremental add of disks to existing RAIDZ by gregmac on Sunday August 20, 2023 @01:06AM (#37196293)
Sorry I said unionfs but it's actually handled by mergerfs [1], and it's all automatic. There are a whole boatload of policies [2] to control writes.

I use "existing path, least free space". Once a path is created, it keeps using it for new files in that path. If it runs out of space, it creates that same path on another drive. If the path exists on both drives for some reason, my rationale is this keeps most of the related files (same path) together on the same drive.

I see there's some newer "most shared path" options I don't remember that might even make more sense for me, so maybe that's something I'll change next time I need to touch it.

[1] https://github.com/trapexit/mergerfs

[2] https://github.com/trapexit/mergerfs#policy-descriptions

Re:OpenZFS – Incremental add of disks to existing RAIDZ by _trapexit on Sunday August 20, 2023 @02:30PM (#37201750)
> If it runs out of space, it creates that same path on another drive.

That's not how it works.

The policy picks what branch to use and then once selected mergerfs will clone the relative path as needed. With "ep" policies it will never select a branch that doesn't have the full relative path. "msp" will always rerun the check one level up in the hierarchy if nothing is found at the current level.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by RulerOf on Monday August 21, 2023 @05:04PM (#37215338)
This is immensely helpful, thanks.

Will MergerFS mount points behave the same as on the host inside of a docker container if passed as a bind mount?

Re:OpenZFS – Incremental add of disks to existing RAIDZ by didntcheck on Sunday August 20, 2023 @06:47AM (#37197895)
I like Snapraid, but there are a few downsides worth mentioning

* Rebuilds are semi-offline, as you said. Almost every other solution, even Unraid, will immediately emulate the data from a failed drive. On Snapraid you have to wait for each file to be restored, and depending on your union setup this may mean you have directories with half the files missing

* Whenever you modify or delete a file between syncs, your parity is now out of sync. This means some other files may fail to restore if you have a failure now. However using more than single parity will make this far less likely to happen. Another way to fix this completely the "snapraid-btrfs" tool, which runs Snapraid on snapshots of independent btrfs disks (somewhat like Synology?), meaning the old data is still available

* It saves almost no file metadata, not even owner and mode. Restored files just use the umask of the user running the restore command. A minor one, but surprisingly annoying

However one big advantage over Unraid it has is how transactional and rigorous it is. A power failure can cause Unraid parity to desync, with no clear way to know which disk is right. Snapraid OTOH is designed to survive interruptions gracefully, and checksums all files so should never accidentally restore corrupt data. And can detect silent drive failure ("bitrot") as a bonus

And for a typical home NAS storing movies and family photos (mostly append-only), those downsides are probably no big deal anyway

Re:OpenZFS – Incremental add of disks to existing RAIDZ by wtallis on Saturday August 19, 2023 @07:17PM (#37194325)
> which rules out all RAID levels except 0, which is nuts.

RAID 1, you mean? Because that way you still have a complete copy of your data if one drive fails.

BTRFS is excellent for the use case of a wide variety of mismatched drives, because it supports adding and removing drives and rebalancing the array. But for the moment only the RAID 1 modes are really trustworthy. I have a NAS consisting of drives whose advertised capacities in GB are: 1000, 1000, 1920, 2000, 2048, 2050, 3840, 3840, 4000. But it started as a pile of infamously unreliable 3TB hard drives.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by hinkley on Saturday August 19, 2023 @08:38PM (#37194825)
Raid 1 does not work with heterogenous disk arrays.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by ilyt on Saturday August 19, 2023 @09:06PM (#37194994)
Eh, it can kinda work. If you have 1, 2,3,4TB drives you can make RAID10 out of first GB, then stitch 2TB RAID0 from second and third and RAID1 it with 2TB out of 4th drive. then raid1 out of remaining GB on 3rd and 4th.

Then glue it together with LVM and hope for best.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by wtallis on Saturday August 19, 2023 @09:22PM (#37195090)
Or you can just put all those devices into one BTRFS RAID 1 filesystem and not have to worry about it.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by ilyt on Sunday August 20, 2023 @12:53PM (#37200863)
BTRFS doesn't exactly had great history with fs bugs
Re:OpenZFS – Incremental add of disks to existing RAIDZ by tatersolid on Sunday August 20, 2023 @06:53PM (#37203704)
Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…
Re:OpenZFS – Incremental add of disks to existing RAIDZ by wtallis on Sunday August 20, 2023 @07:52PM (#37204062)
I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility of BTRFS gives you a lot more options than eg. ZFS for getting out of a tight spot without the full wipe and restore hassle. But both can still leave you stuck, especially if you ignore the need for monitoring and preventive maintenance.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by tatersolid on Monday August 21, 2023 @12:30AM (#37205512)
Disk full is not a “corner case”. It should be one of the first test cases a filesystem test suite should have, and that test should always pass in every configuration.

The fact that total data loss happened regularly on full volumes many years after a “stable” release means BTRFS is either fundamentally flawed in design or incompetently implemented. I have no idea how it was accepted into mainline Linux.

The fact that a certain large NAS vendor used BTRFS by default (without documentation at the time of purchase) has cost my org and others lots of downtime and money.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by wtallis on Monday August 21, 2023 @08:08AM (#37208366)
> Disk full is not a “corner case”.

I didn't say it was. There are plenty of disk full scenarios that can be handled by BTRFS without trouble, especially when adding more disks (even temporarily) is an option. There are ways to get stuck with a full fs, but it's not a guarantee you'll end up in a "wipe and restore from backup" situation. (And to be fair, ZFS also gets very problematic when completely full; this is a general problem for CoW filesystems.)

Re:OpenZFS – Incremental add of disks to existing RAIDZ by ilyt on Monday August 21, 2023 @09:44AM (#37209505)
Sooo basically if you were using XFS you wouldn't need to do anything at all till it got 99% full, and if you got it there you'd just get "disk is full" message instead of some catastrophe? I don't need that kind of stress in my life...

And with plain RAID6 I can still add hard drives if needed. Yeah the case of buying bigger drives is still a bit iffy but btrfs RAID1 gonna lose me more space than RAID6+LVM+xfs anyway...

Re:OpenZFS – Incremental add of disks to existing RAIDZ by hinkley on Saturday August 19, 2023 @10:11PM (#37195378)
There is what a man can do, and what he should do. Nobody normal is going to do any of this.
Re:OpenZFS – Incremental add of disks to existing RAIDZ by wtallis on Saturday August 19, 2023 @09:20PM (#37195075)
BTRFS RAID 1 works just fine with heterogeneous disks with no extra configuration required; I would not have been able to provide examples from experience otherwise. When writing new data, it allocates two blocks on two separate devices, preferring to allocate space on whatever devices have the most free space. This may not be optimal from a performance perspective, but it does mean there's minimal wasted capacity in most configurations. (The RAID1c3 and RAID1c4 modes work the same, but with three or four copies of each block, all on separate devices.)

Were you assuming that BTRFS RAID 1 meant every block of data gets mirrored across every device? I've seen that assumption before, from people who don't realize there are two ways to generalize RAID1 to more than two devices.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by keeperofdakeys on Saturday August 19, 2023 @10:38PM (#37195540)
[delayed]
Re:OpenZFS – Incremental add of disks to existing RAIDZ by wtallis on Saturday August 19, 2023 @10:55PM (#37195633)
> The BTRFS raid1 feature is very badly named,

You're going to have to support that claim a bit better. The core idea of RAID 1 is mirroring data, which BTRFS RAID 1 mode definitely does. Striping is not an essential part of RAID 1 (hence RAID 10), and reading data from two disks in parallel is an optional performance optimization that is not performed by all RAID 1 implementations (but could be implemented for BTRFS RAID 1: https://stackoverflow.com/questions/55408256/btrfs-raid-1-wh... ).

> Plus with mismatched drives, smaller drives go unused unless you write enough data.

Yes, the allocation is suboptimal from a performance perspective, as I've already said. But it is simple and straightforward and reasonably good at avoiding putting you into a situation where manually issuing a rebalance command is necessary. If you do need better performance, there's a RAID 10 mode. But since my NAS is currently a motley pile of SSDs, I don't need to to anything extra to have decent performance.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by keeperofdakeys on Saturday August 19, 2023 @11:59PM (#37195999)
RAID1 is about mirroring disks, BTRFS RAID1 mirrors block groups. Plus traditionally a RAID1 of 3 disks will mirror the same data on all disks, which is different to how the RAID1 mode on BTRFS acts. So the name leads to misunderstandings since it doesn't act like RAID1 at all.

It'd be way easier to talk about if it had a unique name, and you could say "It's like RAID1".

Despite all that I do like the mode, and use it in a few places.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by PlutoIsAPlanet on Sunday August 20, 2023 @06:13AM (#37197736)
> Plus traditionally a RAID1 of 3 disks will mirror the same data on all disks, which is different to how the RAID1 mode on BTRFS acts.

You have raid1c<drive count> mode in BTRFS which does this.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by ilyt on Saturday August 19, 2023 @09:01PM (#37194968)
Unraid "solves" that althought needs some user knowledge (IIRC parity drives must be the biggest one in array and for some reason that isn't automatic)

Ceph actually works quite well for that, althought obviously far more complex than anything for home use. You just tell it "x chunks with N parity" and it will spread it over available drives. Just need more drives than x + n and not too egregious size differences.

> Weighted consistent hashing can handle disparate resources, by assigning more buckets to faster or larger machines. And it can grow and shrink (in a drive array, the two are sequential or simultaneous).

It would need to be more complex than that. Putting chunks on say 2, 6, 8, 12, 22 TB (which might be what you'd get if you just buy "cheapest GB/$" for your NAS over last 10 years!) is more complex than that.

Re:OpenZFS – Incremental add of disks to existing RAIDZ by Tijdreiziger on Saturday August 19, 2023 @09:20PM (#37195074)
Re:OpenZFS – Incremental add of disks to existing RAIDZ by hinkley on Saturday August 19, 2023 @10:10PM (#37195371)
That creates multiple arrays from the same disks. That’s intimately no simpler than buying a string of 2 disk raid enclosures.

Do any of you people actually do UX or DX for a living? This is starting to worry me.