News for hackers. Stuff that matters.
Re:ZFS Is the Best Filesystem For Now by phil21 on Wednesday July 12, 2017 @03:38PM (#14755881)
This is only difficult with ZFS if you care about performance. If you are a typical home file server user, you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well.

There is a huge performance penalty of course, as all the majority of new data will reside on the latest vdevs - but it generally works.

I do agree this is one of the largest drawbacks of ZFS, but very few filesystems get it right.

Re:ZFS Is the Best Filesystem For Now by sfoskett on Wednesday July 12, 2017 @04:35PM (#14756411)
Yeah, as mentioned in the post, you can indeed just add a vdev to the pool whenever you want. But the risk, apart from performance, is data protection If you add one disk, your whole pool is at risk. So of course you have to add two or more disks as a mirror or RAIDZ but lots of home users don't want to do this. They want that Drobo thing where they throw another log on the fire and everything gets rebalanced. It's sad that ZFS can't do this, and that it can't "upgrade" RAID-Z1 to RAID-Z2. It's hard to do, but would be so much nicer.
Re:ZFS Is the Best Filesystem For Now by wmf on Wednesday July 12, 2017 @03:50PM (#14755989)
you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well

As long as they're expensive mirror vdevs, right? My impression is that home users want the efficiency of RAID-6 and they want incremental expansion (regardless of whether this combination is "good for them"). ZFS can't do that.

Re:ZFS Is the Best Filesystem For Now by floatboth on Wednesday July 12, 2017 @05:17PM (#14756775)
You should always use mirrors! http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-... "don’t be greedy. 50% storage efficiency is plenty". Mirrors perform better, perform MUCH better when degraded, rebuild MUCH faster.
Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @01:52AM (#14759017)
Mirrors are also less safe. [0]

For a 6x8TB and assuming a (optimistic) 10^-16 URE, you get 3.5% failure rate for a RAID5 array, 0.7% for a RAID10 array and a 1.06e-08% failure rate for RAID 6.

Why be greedy for all that performance? Most home-grade NAS or even some business-grade NAS isn't used for performance sensitive operations, more like Word Documents and Family Pictures, stuff you don't want to loose.

I'd rather take safety over performance here.

[0]: https://redd.it/6i4n4f

Re:ZFS Is the Best Filesystem For Now by hvidgaard on Thursday July 13, 2017 @09:28AM (#14761008)
Raid is for convinience and performance. It is not and cannot replace backup. In any case, if you want safety, RAID1 is the way forward, and not RAID6. An 8 drive RAID6 with 8TB WD Red NAS drives (URE <1 in 10^14) is virtually guranteed to have at least one read error during a rebuild (if the URE rate is true, which I believe it is not).

Regardless, the determining factor here is how much data do you need to read in the case of a failure to rebuild the array. RAID1 wins every single time because you cannot read less than the single drive you need to replace.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @09:36AM (#14761064)
The URE rate is most likely much lower.

However, the chance of failure for a RAID6 of 100x10TB disks is less than 0.482% after 1'000'000 rebuilds.

RAID1 is space-inefficient, a 100x10TB RAID array might never fail but it has only 10TB of storage space.

A RAID10 Array has a 14% failure chance for just 4x2TB disks using 10^14 failure rates.

RAID1 and RAID10 are definitely not the way forward, it is less secure, something that should be immediately apparent if you read the link in my previous comment.

A 10 Disk RAID6 with 10TB disks is more reliably than a RAID10 by multiple orders of magnitude and more space efficient than a simple RAID1.

Re:ZFS Is the Best Filesystem For Now by hvidgaard on Friday July 14, 2017 @03:07AM (#14768104)
Your math is completely off. a 100x10TB RAID6, with a failed disk need to read 990TB of data to rebuild in the case of a failed disk. With an URE of 1 in 10^14 you will see 79.2 URE events on average if the URE rate is correct (again, I don't believe it is) during single a rebuild - this is the reason no serious engineer recommends a RAID6 for large arrays.

In the case of a RAID1, noone uses 100 mirrored drives. You use RAID10, and in the case of a failed disk, you must read 10TB to recover. With the same URE, we'd see on average 8 URE for every 10 rebuilds, or around 2 orders of magnitude less failure rate compared to the RAID6 example.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Friday July 14, 2017 @03:27AM (#14768172)
Your logic is sadly incorrect.

During a RAID6 rebuild, a URE is non-critical as the Array can recover the data with one lost disk an a URE on any other disk during the stripe rebuild.

The only critical error would be a URE on two disk on the same stripe, 80 URE's during a 990TB rebuild have an amazingly low chance of having two UREs on the same stripe on two seperate disks.

In case of the RAID10, you get 8 URE's over 10 rebuilds, which aren't recoverable unless you have 3 disks. So you'll corrupt data.

edit: URE of 10^14 is what most vendors specify for consumer harddrives, 10^16 is closer to what people encounter in the real world but 10^14 is considered the worst case URE rate.

Re:ZFS Is the Best Filesystem For Now by hvidgaard on Friday July 14, 2017 @07:06AM (#14768888)
Good point with the URE on a RAID6, but that still doesn't make it superiour. The strain of rebuild have been known to kill many arrays, both RAID5 and 6.

URE does not have to corrupt data, if you use a proper filesystem with checksumming such as the ZFS.

When a disk fails, a RAID10 is simply in a far better position as it only have to read a single disk, and it doesn't have any complicated striping to worry about. Just clone a disk.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Friday July 14, 2017 @09:36AM (#14769650)
>URE does not have to corrupt data, if you use a proper filesystem with checksumming such as the ZFS.

No but afaik there is no way to recover data once ZFS has declared it corrupted. (ie, no parity)

>The strain of rebuild have been known to kill many arrays, both RAID5 and 6.

I haven't actually encountered that yet. Despite that, a RAID 6 can loose a disk, so as long as you don't encounter further URE's after loosing another disk, it's fine.

If you're worried about that, go for RAIDZ3 or equivalent. With something like SnapRAID you can even have a RAIDZ6, loosing 6 disks without loosing data. The chances of that happening are relatively low.

>When a disk fails, a RAID10 is simply in a far better position as it only have to read a single disk

A RAID 10 is in no position to recover from URE's once a disk has failed unless you reduce your space efficiency to 33%.

I personally favor not corrupting data over rebuild speeds.

Striping might be complicated but that doesn't make it worse.

It might be acceptable too loose a music file, but once the family image collection gets corrupted or even lost on ZFS because a disk in a RAID 1 encountered a URE, it's personal.

I'd rather life with the thought that even if a disk has a URE, the others can cover for it. Even during a rebuild.

Re:ZFS Is the Best Filesystem For Now by jbronn on Wednesday July 12, 2017 @04:12PM (#14756209)
While it's true you can't expand a vdev, you can always add another vdev to a pool at any redundancy level you desire to expand capacity. For example, you could add a trio of drives as a raidz vdev to a pool with an existing mirror vdev (`zpool add mypool raidz dev1 dev2 dev3`). However, the drawback is that the expanded pool won't be "balanced" unless its datasets are rewritten.
Re:ZFS Is the Best Filesystem For Now by tscs37 on Wednesday July 12, 2017 @04:34PM (#14756399)
Home users would rather just add a single drive instead of having to worry about adding entire RAID setups.
Re:ZFS Is the Best Filesystem For Now by alyandon on Wednesday July 12, 2017 @04:48PM (#14756537)
Exactly that. I can essentially add/remove drives on a whim (for any reasonable definition of "on a whim") using btrfs and run a rebalance command afterward and I'm done.

That, of course, is really not a concern for enterprise use cases.

Re:ZFS Is the Best Filesystem For Now by eriknstr on Wednesday July 12, 2017 @03:45PM (#14755935)
What I'd like to know is, can this issue be resolved in a future version of ZFS or is it too ingrained into the design of ZFS?
Re:ZFS Is the Best Filesystem For Now by wtallis on Wednesday July 12, 2017 @05:04PM (#14756665)
For years the fix ("block pointer rewrite" feature) was promised as coming eventually, but that effort was abandoned. BTRFS will reach ZFS levels of stability before ZFS reaches BTRFS levels of flexibility.
Re:ZFS Is the Best Filesystem For Now by cbhl on Wednesday July 12, 2017 @04:29PM (#14756353)
It would be expensive -- you'd have to "rebalance" the data by copying it to the new volume / away from the old volume, which would take hours or days.
Re:ZFS Is the Best Filesystem For Now by rocqua on Wednesday July 12, 2017 @05:38PM (#14756936)
People who want this feature are mostly frugal. They want it so they can upgrade by just buying a single new disk. That is a rare enough use case that the performance penalty would be worth it.
Re:ZFS Is the Best Filesystem For Now by cbhl on Saturday July 15, 2017 @11:47AM (#14777228)
Frugal enough to shut off their computer when they're not using it (to "save power"), so the rebalancing won't have time to complete in the background?
Re:ZFS Is the Best Filesystem For Now by kalleboo on Wednesday July 12, 2017 @11:57PM (#14758583)
Isn't the fact that Drobo is still around and doing well a testament to people being fine with this tradeoff?
Re:ZFS Is the Best Filesystem For Now by problems on Wednesday July 12, 2017 @04:43PM (#14756489)
Correct me if I'm wrong, but I can't add vdevs to a RAIDZ (RAID5/6ish) system in ZFS, which are probably the most common configurations for home NAS systems.

I'd love to run ZFS, but I can't because I need to be able to add more drives as I buy them.

Re:ZFS Is the Best Filesystem For Now by problems on Thursday July 13, 2017 @05:02PM (#14765204)
I've been reading more into it and it seems like back in 2008 they came up with an algorithm to do this: https://blogs.oracle.com/ahl/expand-o-matic-raid-z

But it doesn't look like there's been any movement on an implementation and it seems like it's high effort and mostly home users who want this, not enterprises who might be willing to pay for.

Ah well, guess I'll stick with mdraid for now.

Re:ZFS Is the Best Filesystem For Now by Terribledactyl on Wednesday July 12, 2017 @05:08PM (#14756698)
correct, the number of devices in a raidz1/2/3 vdev cannot change. However you can replace all of the drives one at a time with higher capacity drives and most zfs installs will grow your vdev.

You could also add another raidz vdev if you have the space in the server, eventually they should level out depending on workload.

"Best" solution, read most expensive, would be to copy the data over to a new server you've configured for the new load/capacity.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @01:55AM (#14759036)
Adding 3 drives at a time is expensive and a non-option for a home NAS. As is "just copy it to a new server".

Both have high investment costs for something a plain mdRAID, snapRAID or LVM RAID can achieve far simpler with better results.

Re:ZFS Is the Best Filesystem For Now by snuxoll on Thursday July 13, 2017 @02:03PM (#14763613)
I add two at a time without much problem, mind you I have a rather extreme setup with an external 12-bay enclosure but I feel most people building a NAS instead of buying an off-the-shelf system from Synology/Drobo aren't looking to cut costs or corners.

A 4TB Seagate IronWolf drive costs $129 off Amazon, buying two to add as a new mirrored vdev to my TrueNAS box isn't outrageous.

Re:ZFS Is the Best Filesystem For Now by problems on Thursday July 13, 2017 @02:10PM (#14763687)
You're limited then to only mirroring, you can't do RAID6 or similar on a massive array where you're only giving up 2/10 drives space but can have a 2 disk failure. Seems like a lot of wasted disks when the only thing I'm dependent on the NAS for is not losing my TV collection. Right now I know if that if I go out and buy 2 more 4 TB disk I get 8 TB more in my NAS and at the price of an only slightly increased risk of having more than 2 drives fail at once. That's probably my favorite feature of RAID6 for a home NAS.

I actually went custom because those off-the-shelf boxes are either very expensive or have weak CPUs so can't be used for video transcoding very well - it's cheaper to build it yourself, much cheaper if you already have old hardware to dedicate to the task.

Re:ZFS Is the Best Filesystem For Now by snuxoll on Thursday July 13, 2017 @02:25PM (#14763834)
Different use cases I suppose, my FreeNAS box stores my video collection and all the usual stuff but I've also got all the VM's that run my home network stored on it - performance+resliver times are a lot more important to me than storage efficiency.
Re:ZFS Is the Best Filesystem For Now by problems on Thursday July 13, 2017 @02:48PM (#14764006)
And your network-running VMs have high disk IO requirements?
Re:ZFS Is the Best Filesystem For Now by snuxoll on Thursday July 13, 2017 @05:05PM (#14765225)
All of my virtual disks are hosted off my FreeNAS, I've got a direct 10GbE link between it and my oVirt host - raw throughput isn't so much the issue most of the time as IOPS are, I've got a local gitlab instance, OpenShift, some PostgreSQL databases, etc. and they like to hammer the crap out of my storage when in use.

Having an L2ARC helps out quite a bit, but only having 32GB of memory and wanting to keep most of it for the L1ARC means I still hit my spinning disks regularly (and mirrored vdev's help read IOPS tremendously in this case).

Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @02:09PM (#14763682)
Reducing the cost of a custom system isn't irrational.

My own personal budget is very limited, buying two IronWolf HDDs in this case is easily a good chunk of my monthly income. If I can build a NAS that can expand with single drives as needed, it's more cost effective for me.

And I imagine a lot of others have the same problem.

In the end, by buying 2 drives when a single drive could have solved the problem equally well: expanding your space by 4TB, is wasting money. Period.