News for hackers. Stuff that matters.
Re:ZFS Is the Best Filesystem For Now by alyandon on Wednesday July 12, 2017 @03:13PM (#14755643)

  "Once you build a ZFS volume, it’s pretty much fixed for life."
The ease of growing/shrinking existing volumes and adding/removing storage is why I made the decision to go with btrfs when I rebuilt my home file server.
Re:ZFS Is the Best Filesystem For Now by bitL on Wednesday July 12, 2017 @03:48PM (#14755964)
I hope you managed to use the fixed BTRFS unlike me that built home NAS with tens of terrabytes on top of BTRFS and then year later learned that the version I have was broken and the fix is not applicable to a running cluster... So I have to buy another batch of HDDs, install fixed BTRFS (who knows what other unknown issue awaits me in the darkness?) and copy over everything...
Re:ZFS Is the Best Filesystem For Now by ioquatix on Wednesday July 12, 2017 @08:37PM (#14757907)
This advice isn't actually totally correct.

If you build a volume, you can add more zdevs to it.

My suggestion is to avoid RAIDZx and go with mirrors. To add more storage, just add another pair of drives and add it to your pool.

Here is what my main `tank` looks like:

  pool: tank
  state: ONLINE
  scan: scrub repaired 0B in 4h59m with 0 errors on Tue Jul  4 12:47:02 2017
  config:

  NAME                            STATE     READ WRITE CKSUM
  tank                            ONLINE       0     0     0
  	mirror-0                      ONLINE       0     0     0
  		wwn-0x50014ee20eba1695      ONLINE       0     0     0
  		wwn-0x50014ee20eba337e      ONLINE       0     0     0
  	mirror-1                      ONLINE       0     0     0
  		wwn-0x50014ee2640efb3a      ONLINE       0     0     0
  		wwn-0x50014ee2b964ef3f      ONLINE       0     0     0
  	mirror-2                      ONLINE       0     0     0
  		wwn-0x50014ee2b964f2d5      ONLINE       0     0     0
  		wwn-0x50014ee2b964f4f4      ONLINE       0     0     0
  cache
  	wwn-0x5002538d704e9ff1-part4  ONLINE       0     0     0

  errors: No known data errors
You can simply add another mirrored pair and get more storage. Or, you can update an existing mirror by adding new drives, then removing the old ones. Mirroring has better performance and IOPs than striped raid too. If you feel that mirrors might be less reliable than RAIDZx, I'd suggest you read up about it - generally mirrors are more reliable (and you could mirror 3 drives if you liked, get more throughput and IOPs - or have a hot spare).
Re:ZFS Is the Best Filesystem For Now by gaadd33 on Wednesday July 12, 2017 @10:58PM (#14758411)
As you add mirrored pairs your data isn't evenly distributed across all stripes is it?
Re:ZFS Is the Best Filesystem For Now by ioquatix on Wednesday July 12, 2017 @11:31PM (#14758518)
The simple answer to your question is no.

The more complex answer is that over time as you write more data, ZFS will re-balance the distribution of data over the available zdevs.

Re:ZFS Is the Best Filesystem For Now by takeda on Thursday July 13, 2017 @11:27PM (#14767216)
I think the concern is that if you have two disks die it is not a big deal, unless those two disk happen to be of the same pair[1], then most likely all data is lost.

[1] if you're not careful, this is actually very likely to happen. If you purchase 2 disks at the same time, and they are under the same usage patterns (which they will be when working in a pair), under the same temperature then they are very likely to die at the same time.

Re:ZFS Is the Best Filesystem For Now by problems on Wednesday July 12, 2017 @04:42PM (#14756473)
Btrfs is a mess too, especially with its metadata system that you have to manually resize constantly, data-loss bugs and fsck problems. I've had nothing but problems with it and will never use it on a production system.

LVM on the other hand makes it easy to do snapshotting, RAID, volume resizing, adding more drives, etc - at the cost of some performance in some cases. For the most part, it's the best trade off available if expandability is a requirement and you can't get away with simple mdraid.

Re:ZFS Is the Best Filesystem For Now by alyandon on Wednesday July 12, 2017 @07:28PM (#14757560)
btrfs has all those features as well and my original setup was mdadm raid 5 + LVM + ext4.

Regardless, I really haven't had any operational problems with btrfs for my 3 TB or so of data and when it has managed to get wedged because it couldn't allocate more space to the metadata pool during the initial import of data from my old array I fixed it with a simple rebalance command.

A cron set up to run a minor rebalance weekly helps ensure you never run into that situation in practice and I've not lost any data so for now I'm comfortable using btrfs as my primary filesystem.

I am a little concerned about the longevity of btrfs in general though because it hasn't been receiving a lot of development work lately.

Re:ZFS Is the Best Filesystem For Now by cmurf on Wednesday July 12, 2017 @06:58PM (#14757407)
>its metadata system that you have to manually resize constantly

?

Doesn't ring a bell.

A lot of problems with Btrfs sounds like hardware problems of one sort or another. If it's a legit bug the only way such things get fixed is to report it to the developers <[email protected]> with complete logs and system information. Did you?

The fsck is definitely hit or miss, but my perspective is the emphasis is on fixing bugs that obviate file system problems in the first place. The reality is an offline fsck for large file systems is just not scalable, so the best bang for the buck is bug squashing.

And there's a tons of that happening.

For the initial 4.12 pull (now done, and probably had a few dozen changes during rc's) 40 files changed, 1629 insertions(+), 834 deletions(-) https://lkml.org/lkml/2017/5/9/510

For the initial 4.13 pull 47 files changed, 1707 insertions(+), 1400 deletions(-) https://lkml.org/lkml/2017/7/4/436

Re:ZFS Is the Best Filesystem For Now by rrauenza on Wednesday July 12, 2017 @08:49PM (#14757973)
He may be talking about the need to rebalance ... I run a btrfs balance every month to make sure I don't run out of space again.

It happened to me while on vacation -- I only had satellite internet and had to fix it over ssh at single characters per second.

It might have to do with a bug I saw mentioned on the list that referred to the free space map getting corrupted (but easily fixed with a -oremount,clear_cache). So I also run that monthly as well.

I've been too afraid to remove the cron job even though I moved my CentOS7 to the "mainline" kernel rpms from http://elrepo.org

Also, I only trust RAID1 + Crashplan.

Re:ZFS Is the Best Filesystem For Now by ThatPlayer on Wednesday July 12, 2017 @03:17PM (#14755674)
I'm also using Btrfs for my home file server, but Btrfs's current RAID5/6 status leaves a lot to be desired. I'm running Btrfs on top of mdadm right now because of that.
Re:ZFS Is the Best Filesystem For Now by sheepdestroyer on Wednesday July 12, 2017 @03:24PM (#14755745)
Should be "mostly" resolved from kernel 4.12 : https://www.phoronix.com/scan.php?page=news_item&px=Linux-4....
Re:ZFS Is the Best Filesystem For Now by pantalaimon on Wednesday July 12, 2017 @03:35PM (#14755852)
It used to be "usable for most purposes" at some point already, before those scary data corruption bugs were discovered.

https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30...

Re:ZFS Is the Best Filesystem For Now by alyandon on Wednesday July 12, 2017 @03:30PM (#14755805)
My last file server used mdadm raid 5 across 4 drives and I just couldn't take the write performance loss anymore so I switched to btrfs raid1(ish) block duplication.
Re:ZFS Is the Best Filesystem For Now by phil21 on Wednesday July 12, 2017 @03:38PM (#14755881)
This is only difficult with ZFS if you care about performance. If you are a typical home file server user, you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well.

There is a huge performance penalty of course, as all the majority of new data will reside on the latest vdevs - but it generally works.

I do agree this is one of the largest drawbacks of ZFS, but very few filesystems get it right.

Re:ZFS Is the Best Filesystem For Now by sfoskett on Wednesday July 12, 2017 @04:35PM (#14756411)
Yeah, as mentioned in the post, you can indeed just add a vdev to the pool whenever you want. But the risk, apart from performance, is data protection If you add one disk, your whole pool is at risk. So of course you have to add two or more disks as a mirror or RAIDZ but lots of home users don't want to do this. They want that Drobo thing where they throw another log on the fire and everything gets rebalanced. It's sad that ZFS can't do this, and that it can't "upgrade" RAID-Z1 to RAID-Z2. It's hard to do, but would be so much nicer.
Re:ZFS Is the Best Filesystem For Now by wmf on Wednesday July 12, 2017 @03:50PM (#14755989)
you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well

As long as they're expensive mirror vdevs, right? My impression is that home users want the efficiency of RAID-6 and they want incremental expansion (regardless of whether this combination is "good for them"). ZFS can't do that.

Re:ZFS Is the Best Filesystem For Now by floatboth on Wednesday July 12, 2017 @05:17PM (#14756775)
You should always use mirrors! http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-... "don’t be greedy. 50% storage efficiency is plenty". Mirrors perform better, perform MUCH better when degraded, rebuild MUCH faster.
Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @01:52AM (#14759017)
Mirrors are also less safe. [0]

For a 6x8TB and assuming a (optimistic) 10^-16 URE, you get 3.5% failure rate for a RAID5 array, 0.7% for a RAID10 array and a 1.06e-08% failure rate for RAID 6.

Why be greedy for all that performance? Most home-grade NAS or even some business-grade NAS isn't used for performance sensitive operations, more like Word Documents and Family Pictures, stuff you don't want to loose.

I'd rather take safety over performance here.

[0]: https://redd.it/6i4n4f

Re:ZFS Is the Best Filesystem For Now by hvidgaard on Thursday July 13, 2017 @09:28AM (#14761008)
Raid is for convinience and performance. It is not and cannot replace backup. In any case, if you want safety, RAID1 is the way forward, and not RAID6. An 8 drive RAID6 with 8TB WD Red NAS drives (URE <1 in 10^14) is virtually guranteed to have at least one read error during a rebuild (if the URE rate is true, which I believe it is not).

Regardless, the determining factor here is how much data do you need to read in the case of a failure to rebuild the array. RAID1 wins every single time because you cannot read less than the single drive you need to replace.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @09:36AM (#14761064)
The URE rate is most likely much lower.

However, the chance of failure for a RAID6 of 100x10TB disks is less than 0.482% after 1'000'000 rebuilds.

RAID1 is space-inefficient, a 100x10TB RAID array might never fail but it has only 10TB of storage space.

A RAID10 Array has a 14% failure chance for just 4x2TB disks using 10^14 failure rates.

RAID1 and RAID10 are definitely not the way forward, it is less secure, something that should be immediately apparent if you read the link in my previous comment.

A 10 Disk RAID6 with 10TB disks is more reliably than a RAID10 by multiple orders of magnitude and more space efficient than a simple RAID1.

Re:ZFS Is the Best Filesystem For Now by hvidgaard on Friday July 14, 2017 @03:07AM (#14768104)
Your math is completely off. a 100x10TB RAID6, with a failed disk need to read 990TB of data to rebuild in the case of a failed disk. With an URE of 1 in 10^14 you will see 79.2 URE events on average if the URE rate is correct (again, I don't believe it is) during single a rebuild - this is the reason no serious engineer recommends a RAID6 for large arrays.

In the case of a RAID1, noone uses 100 mirrored drives. You use RAID10, and in the case of a failed disk, you must read 10TB to recover. With the same URE, we'd see on average 8 URE for every 10 rebuilds, or around 2 orders of magnitude less failure rate compared to the RAID6 example.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Friday July 14, 2017 @03:27AM (#14768172)
Your logic is sadly incorrect.

During a RAID6 rebuild, a URE is non-critical as the Array can recover the data with one lost disk an a URE on any other disk during the stripe rebuild.

The only critical error would be a URE on two disk on the same stripe, 80 URE's during a 990TB rebuild have an amazingly low chance of having two UREs on the same stripe on two seperate disks.

In case of the RAID10, you get 8 URE's over 10 rebuilds, which aren't recoverable unless you have 3 disks. So you'll corrupt data.

edit: URE of 10^14 is what most vendors specify for consumer harddrives, 10^16 is closer to what people encounter in the real world but 10^14 is considered the worst case URE rate.

Re:ZFS Is the Best Filesystem For Now by hvidgaard on Friday July 14, 2017 @07:06AM (#14768888)
Good point with the URE on a RAID6, but that still doesn't make it superiour. The strain of rebuild have been known to kill many arrays, both RAID5 and 6.

URE does not have to corrupt data, if you use a proper filesystem with checksumming such as the ZFS.

When a disk fails, a RAID10 is simply in a far better position as it only have to read a single disk, and it doesn't have any complicated striping to worry about. Just clone a disk.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Friday July 14, 2017 @09:36AM (#14769650)
>URE does not have to corrupt data, if you use a proper filesystem with checksumming such as the ZFS.

No but afaik there is no way to recover data once ZFS has declared it corrupted. (ie, no parity)

>The strain of rebuild have been known to kill many arrays, both RAID5 and 6.

I haven't actually encountered that yet. Despite that, a RAID 6 can loose a disk, so as long as you don't encounter further URE's after loosing another disk, it's fine.

If you're worried about that, go for RAIDZ3 or equivalent. With something like SnapRAID you can even have a RAIDZ6, loosing 6 disks without loosing data. The chances of that happening are relatively low.

>When a disk fails, a RAID10 is simply in a far better position as it only have to read a single disk

A RAID 10 is in no position to recover from URE's once a disk has failed unless you reduce your space efficiency to 33%.

I personally favor not corrupting data over rebuild speeds.

Striping might be complicated but that doesn't make it worse.

It might be acceptable too loose a music file, but once the family image collection gets corrupted or even lost on ZFS because a disk in a RAID 1 encountered a URE, it's personal.

I'd rather life with the thought that even if a disk has a URE, the others can cover for it. Even during a rebuild.

Re:ZFS Is the Best Filesystem For Now by jbronn on Wednesday July 12, 2017 @04:12PM (#14756209)
While it's true you can't expand a vdev, you can always add another vdev to a pool at any redundancy level you desire to expand capacity. For example, you could add a trio of drives as a raidz vdev to a pool with an existing mirror vdev (`zpool add mypool raidz dev1 dev2 dev3`). However, the drawback is that the expanded pool won't be "balanced" unless its datasets are rewritten.
Re:ZFS Is the Best Filesystem For Now by tscs37 on Wednesday July 12, 2017 @04:34PM (#14756399)
Home users would rather just add a single drive instead of having to worry about adding entire RAID setups.
Re:ZFS Is the Best Filesystem For Now by alyandon on Wednesday July 12, 2017 @04:48PM (#14756537)
Exactly that. I can essentially add/remove drives on a whim (for any reasonable definition of "on a whim") using btrfs and run a rebalance command afterward and I'm done.

That, of course, is really not a concern for enterprise use cases.

Re:ZFS Is the Best Filesystem For Now by eriknstr on Wednesday July 12, 2017 @03:45PM (#14755935)
What I'd like to know is, can this issue be resolved in a future version of ZFS or is it too ingrained into the design of ZFS?
Re:ZFS Is the Best Filesystem For Now by wtallis on Wednesday July 12, 2017 @05:04PM (#14756665)
For years the fix ("block pointer rewrite" feature) was promised as coming eventually, but that effort was abandoned. BTRFS will reach ZFS levels of stability before ZFS reaches BTRFS levels of flexibility.
Re:ZFS Is the Best Filesystem For Now by cbhl on Wednesday July 12, 2017 @04:29PM (#14756353)
It would be expensive -- you'd have to "rebalance" the data by copying it to the new volume / away from the old volume, which would take hours or days.
Re:ZFS Is the Best Filesystem For Now by rocqua on Wednesday July 12, 2017 @05:38PM (#14756936)
People who want this feature are mostly frugal. They want it so they can upgrade by just buying a single new disk. That is a rare enough use case that the performance penalty would be worth it.
Re:ZFS Is the Best Filesystem For Now by cbhl on Saturday July 15, 2017 @11:47AM (#14777228)
Frugal enough to shut off their computer when they're not using it (to "save power"), so the rebalancing won't have time to complete in the background?
Re:ZFS Is the Best Filesystem For Now by kalleboo on Wednesday July 12, 2017 @11:57PM (#14758583)
Isn't the fact that Drobo is still around and doing well a testament to people being fine with this tradeoff?
Re:ZFS Is the Best Filesystem For Now by problems on Wednesday July 12, 2017 @04:43PM (#14756489)
Correct me if I'm wrong, but I can't add vdevs to a RAIDZ (RAID5/6ish) system in ZFS, which are probably the most common configurations for home NAS systems.

I'd love to run ZFS, but I can't because I need to be able to add more drives as I buy them.

Re:ZFS Is the Best Filesystem For Now by problems on Thursday July 13, 2017 @05:02PM (#14765204)
I've been reading more into it and it seems like back in 2008 they came up with an algorithm to do this: https://blogs.oracle.com/ahl/expand-o-matic-raid-z

But it doesn't look like there's been any movement on an implementation and it seems like it's high effort and mostly home users who want this, not enterprises who might be willing to pay for.

Ah well, guess I'll stick with mdraid for now.

Re:ZFS Is the Best Filesystem For Now by Terribledactyl on Wednesday July 12, 2017 @05:08PM (#14756698)
correct, the number of devices in a raidz1/2/3 vdev cannot change. However you can replace all of the drives one at a time with higher capacity drives and most zfs installs will grow your vdev.

You could also add another raidz vdev if you have the space in the server, eventually they should level out depending on workload.

"Best" solution, read most expensive, would be to copy the data over to a new server you've configured for the new load/capacity.

Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @01:55AM (#14759036)
Adding 3 drives at a time is expensive and a non-option for a home NAS. As is "just copy it to a new server".

Both have high investment costs for something a plain mdRAID, snapRAID or LVM RAID can achieve far simpler with better results.

Re:ZFS Is the Best Filesystem For Now by snuxoll on Thursday July 13, 2017 @02:03PM (#14763613)
I add two at a time without much problem, mind you I have a rather extreme setup with an external 12-bay enclosure but I feel most people building a NAS instead of buying an off-the-shelf system from Synology/Drobo aren't looking to cut costs or corners.

A 4TB Seagate IronWolf drive costs $129 off Amazon, buying two to add as a new mirrored vdev to my TrueNAS box isn't outrageous.

Re:ZFS Is the Best Filesystem For Now by problems on Thursday July 13, 2017 @02:10PM (#14763687)
You're limited then to only mirroring, you can't do RAID6 or similar on a massive array where you're only giving up 2/10 drives space but can have a 2 disk failure. Seems like a lot of wasted disks when the only thing I'm dependent on the NAS for is not losing my TV collection. Right now I know if that if I go out and buy 2 more 4 TB disk I get 8 TB more in my NAS and at the price of an only slightly increased risk of having more than 2 drives fail at once. That's probably my favorite feature of RAID6 for a home NAS.

I actually went custom because those off-the-shelf boxes are either very expensive or have weak CPUs so can't be used for video transcoding very well - it's cheaper to build it yourself, much cheaper if you already have old hardware to dedicate to the task.

Re:ZFS Is the Best Filesystem For Now by snuxoll on Thursday July 13, 2017 @02:25PM (#14763834)
Different use cases I suppose, my FreeNAS box stores my video collection and all the usual stuff but I've also got all the VM's that run my home network stored on it - performance+resliver times are a lot more important to me than storage efficiency.
Re:ZFS Is the Best Filesystem For Now by problems on Thursday July 13, 2017 @02:48PM (#14764006)
And your network-running VMs have high disk IO requirements?
Re:ZFS Is the Best Filesystem For Now by snuxoll on Thursday July 13, 2017 @05:05PM (#14765225)
All of my virtual disks are hosted off my FreeNAS, I've got a direct 10GbE link between it and my oVirt host - raw throughput isn't so much the issue most of the time as IOPS are, I've got a local gitlab instance, OpenShift, some PostgreSQL databases, etc. and they like to hammer the crap out of my storage when in use.

Having an L2ARC helps out quite a bit, but only having 32GB of memory and wanting to keep most of it for the L1ARC means I still hit my spinning disks regularly (and mirrored vdev's help read IOPS tremendously in this case).

Re:ZFS Is the Best Filesystem For Now by tscs37 on Thursday July 13, 2017 @02:09PM (#14763682)
Reducing the cost of a custom system isn't irrational.

My own personal budget is very limited, buying two IronWolf HDDs in this case is easily a good chunk of my monthly income. If I can build a NAS that can expand with single drives as needed, it's more cost effective for me.

And I imagine a lot of others have the same problem.

In the end, by buying 2 drives when a single drive could have solved the problem equally well: expanding your space by 4TB, is wasting money. Period.