News for hackers. Stuff that matters.
Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @04:23AM (#25158761)
What are the limits of ZFS, can it scale up to the petabyte range like Ceph?
Re:An Introduction to ZFS by Tuna-Fish on Friday November 20, 2020 @06:22AM (#25159391)
ZFS can scale a lot further than Ceph. The limits were designed to be large enough to be never encountered in practice. Not just large enough to be never encountered by the people working on it, but large enough that it is not possible to fit a filesystem that needs more on earth, no matter how good your technology is.

The only limits that can be reached is that the maximum size of a single file is 2^64 bytes, or 16 exabytes, and the maximum amount of files in a single directory is 2^48, or 281 trillion. The other limits are large enough that to reach them would generally require more energy than it would take to literally boil the oceans.

Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @07:29AM (#25159799)
I mean practically speaking. Lets say I actually were to buy 100 x 10TB drives. I know that I can span a ceph cluster across those drives (e.g. CERN has created some 30PB cluster in the past). Can I use 100 drives in a RAID-Z?

I'm asking because the main advantage I can see for a zettabyte file system (ZFS) is at larger scale, thus I'm interested in how to actually use it e.g. for a server setup of a company that needs to save data on many disks.

Re:An Introduction to ZFS by magicalhippo on Friday November 20, 2020 @09:40AM (#25160820)
> Can I use 100 drives in a RAID-Z

AFAIK yes you could, but you wouldn't want to put them in a single RAID-Z(1,2 or 3) VDEV, but rather multiple VDEVs. This is because I/O operations scale with the number of VDEVs rather than number of disks[1]. So there's a tradeoff between space efficiency and IOPS. But AFAIK you absolutely could put 100 disks in a single RAID-Z VDEV.

On the mailing lists there's frequently people posting with that or more disks in a single pool (split across multiple VDEVs).

[1]: https://www.delphix.com/blog/delphix-engineering/zfs-raidz-s...

Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @09:48AM (#25160905)
Let's say I build such a system and in 2 years from now, I want to add another 20 disks, is that possible too?
Re:An Introduction to ZFS by magicalhippo on Friday November 20, 2020 @09:57AM (#25160994)
That's the easy thing with ZFS, just add the disks as new VDEVs.

So just to clarify. A ZFS pool stores data in VDEVs. A single VDEV can be:

- a single disk (no redundancy, but same error checking)

- N disks in a N-way mirror

- N disks in a RAID-Z (single for disk parity, min 3 disks)

- RAID-Z2 with 2 disk parity

- RAID-Z3 with 3 disk parity (much slower than other two RAID-Z's due to complex math)

You can always _add_ VDEVs to a pool, and ZFS will start using it. It will try to be clever about it, so for example if existing VDEVs are quite full, it will redirect most of the writes to the new VDEV.

What you cannot do (yet) is _remove_ a VDEV.

It's also possible to replace _all_ the disks in a single VDEV to larger disks in turn (important!), and once the final disk is replaced the VDEV will suddenly appear to have the added capacity.

Re:An Introduction to ZFS by iooi on Friday November 20, 2020 @12:34PM (#25162750)
> That's the easy thing with ZFS, just add the disks as new VDEVs.

Careful though, depending on how you're setting up the VDEVs this could introduce fragility to the pool. For example, if you have a raidz2 partition and then add a single disk vdev, that single disk going down would make the entire pool unavailable.

Re:An Introduction to ZFS by magicalhippo on Friday November 20, 2020 @03:13PM (#25164428)
Good point!

Only way to recover from this would be to either add another disk turning the single-disk VDEV to a mirror, or send/receive the entire pool.

Re:An Introduction to ZFS by rewtraw on Wednesday November 25, 2020 @03:39PM (#25214041)
It is actually now possible to remove a vdev. It's a new feature, not sure if it works for all RAID types, but I just removed a mirror from my pool without any issue. It migrated data off that vdev and into the rest of the pool just like a resilver.
Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @10:10AM (#25161152)
> That's the easy thing with ZFS, just add the disks as new VDEVs

What are the drawbacks of this approach and why can't I just add additional drives to an existing VDEV?

Re:An Introduction to ZFS by magicalhippo on Friday November 20, 2020 @10:49AM (#25161534)
Well, you can add drives to _some_ VDEV types. For a single drive VDEV you can add a drive and it will become a mirror. With a 2-way mirror you can add a drive and it will become a 3-way mirror.

However for the RAID-Z variants it's more difficult, because of how the parity calculations are done and how the parity is stored. If you have a 5 drive RAID-Z, you can store up to 4 blocks of data for each parity block. With a 6 drive RAID-Z you can store up to 5 blocks of data for each parity block.

So if you add a drive to a RAID-Z vdev, the system suddenly needs to keep track of the stripe width that was used when the data was written.

In addition it's the issue of _where_ the data is stored. If your RAID-Z VDEV is mostly full and you were to add a new drive to it (not that you currently can), then you might not be able to use all of the capacity without further ado.

This is because you can only store one block per stripe on the new drive for redundancy to work, a RAID-Z1 can only recover from the loss of one block per stripe. The rest of the blocks in the stripe has to be distributed on the other drives.

However, there is work in progress[1] to support RAID-Z expansion.

That said, RAID-Z expansion is for expanding one drive at a time. If you're adding 20 disks in one go, do it as a new VDEV.

[1]: https://github.com/openzfs/zfs/pull/8853

Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @11:23AM (#25161883)
Is there any official statement from the ZFS maintainers on when we can expect the Pull Request (RAID-Z expansion) to be merged?
Re:An Introduction to ZFS by magicalhippo on Friday November 20, 2020 @11:45AM (#25162155)
Not really, but as you can see from the PR[1] there is work being done.

The ZFS devs don't like to push unstable features, so I expect we're still quite a way off.

[1]: https://github.com/openzfs/zfs/pull/8853

Re:An Introduction to ZFS by secabeen on Friday November 20, 2020 @11:51AM (#25162235)
Years. It'll be years.
Re:An Introduction to ZFS by liuliu on Friday November 20, 2020 @10:46AM (#25161513)
For RaidZ cases, you need to rebalance all the strips, which is non-trivial to do online (you have to make sure if the power cut off during the rebalancing, you don't lose the data). Without rebalancing, I think it can end up with weird corner cases (for example, in RaidZ2 case, if you have the rest 5 disks full and added 1 more disk, you cannot write data into the pool because there is no place for parity. It is not a problem if the disks are balanced).
Re:An Introduction to ZFS by gh02t on Friday November 20, 2020 @09:06AM (#25160523)
ZFS can use that many drives, but it's not a distributed filesystem so it's intended to use within a single server. For 100 disks (or some arbitrarily large number) I'd assume you want something distributed, in which case ZFS could serve as the underlying FS on each node with e.g. Gluster or Ceph. ZFS as the backing store for Gluster is fairly common, for example.
Re:An Introduction to ZFS by secabeen on Friday November 20, 2020 @11:52AM (#25162242)
I have two systems with 140 drives in a single ZFS pool. (14 VDEVS of 10 drives each). They work great, and we haven't had any issues with them.
Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @12:04PM (#25162383)
What type of connection do these system have (e.g. 10GbE) and how much latency? Would it be possible to span a ZFS Pool across multiple regions (us, europe, ...)?
Re:An Introduction to ZFS by secabeen on Friday November 20, 2020 @12:55PM (#25162981)
The two systems are in different buildings on campus. Each system (with 140) drives is a single unit, with 60-drive JBODs running on SAS-external connections.

If you want to span a pool across multiple regions, you'll want a distributed file system on top of your ZFS to manage that. Something like Lustre or Ceph. It would still be very challenging, though.

Re:An Introduction to ZFS by rsync on Friday November 20, 2020 @01:13PM (#25163147)
"What are the limits of ZFS, can it scale up to the petabyte range like Ceph?"

We (rsync.net) have scaled zpools to petabyte range.

A current, example configuration would be:

- 60-drive JBODs - 15 drive raidz3 vdevs, four per JBOD - 16TB SAS drives

That ends up being ~192TB per vdev, 768 TB per JBOD ... and if you span a pool across two JBODs, you have ~1.5 PB.

I should note that what makes it possible to sleep at night with such a configuration is the fact that raidz3 exists. If not for that, I would not configure 15 drive vdevs with jut raidz2 ("raid6") protection.

Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @01:42PM (#25163466)
Has rsync.net ever considered Ceph and if yes, why decided against it in favour of ZFS?
Re:An Introduction to ZFS by rsync on Friday November 20, 2020 @02:47PM (#25164144)
No, we did not ever consider ceph.

rsync.net architecture is purely FreeBSD and there was just a very good fit - and roadmap - from UFS2 which we used from 2001-2012 and ZFS which we have used since.

Also, for better or worse, rsync.net is about UNIX and ZFS is very unixy. We think in filesystems and files and directories and ZFS let's us keep that set of abstractions.

Re:An Introduction to ZFS by gbrown_ on Friday November 20, 2020 @04:30AM (#25158796)
Ceph is a distributed storage system whereas ZFS is not so you can't really compare the two by the capacity people run them at.
Re:An Introduction to ZFS by tutfbhuf on Friday November 20, 2020 @04:57AM (#25158955)
Okay, if I'm not allowed to compare them directly, what are their different use cases?
Re:An Introduction to ZFS by hosteur on Friday November 20, 2020 @04:32AM (#25158815)
I believe the z in zfs is for zettabyte