I’m not a filesystem admin, but we at LLNL use OpenZFS as the storage layer for all of our Lustre file systems in production, including using raid-z for resilience in each pool (order 100 disks each), and have for most of a decade. That combined with improvements in Lustre have taken the rate of data loss or need to clear large scale shared file systems down to nearly zero. There’s a reason we spend as many engineer hours as we do maintaining it, it’s worth it.

LLNL openzfs project: https://computing.llnl.gov/projects/openzfs Old presentation from intel with info on what was one of our bigger deployments in 2016 (~50pb): https://www.intel.com/content/dam/www/public/us/en/documents...

If I'm not mistaken the linux port of ZFS that later became OpenZFS started at LLNL and was a port from FreeBSD (it may have been release ~9).

I believe it was called ZFS On Linux or something like that.

Nice how things have evolved: from FreeBSD to linux and back. In my mind this has always been a very inspiring example of a public institution working for the public good.

FreeBSD had its own ZFS port.

ZoL, if my ancient memory serves, was at LLNL, not based on the FreeBSD port (if you go _very_ far back in the commit history you can see Brian rebasing against OpenSolaris revisions), but like 2 or 3 different orgs originally announced Linux ports at the same time and then all pooled together, since originally only one of the three was going to have a POSIX layer (the other two didn't need a working POSIX filesystem layer). (I'm not actually sure how much came of this collaboration, I just remember being very amused when within the span of a week or two, three different orgs announced ports, looked at each other, and went "...wait.")

Then for a while people developed on either the FreeBSD port, the illumos fork called OpenZFS, or the Linux port, but because (among other reasons) a bunch of development kept happening on the Linux port, it became the defacto upstream and got renamed "OpenZFS", and then FreeBSD more or less got a fresh port from the OpenZFS codebase that is now what it's based on.

The macOS port got a fresh sync against that codebase recently and is slowly trying to merge in, and then from there, ???

We use ZFS in production for the non-content filesystems on a large, ever increasing, percentage of our Netflix Open Connect CDN nodes, replacing geom's gmirror. We had one gotcha (caught on a very limited set of canaries) where a buggy bootloader crashed part-way through boot, leaving the ZFS "bootonce" stuff in a funky state requiring manual recovery (the nodes with gmirror were fine, and fell back to the old image without fuss). This has since been fixed.

Note that we do not use ZFS for content, since it is incompatible with efficient use of sendfile (both because there is no async handler for ZFS, so no async sendfile, and because the ARC is not integrated with the page cache, so content would require an extra memory copy to be served).

Is integrating ARC with the page cache a lost cause? If not, may be Netflix can fund it!

I would expect page cache to be one of those platform-dependent things that prevent OpenZFS from doing that. Especially on Linux, and especially because AFAIK the Linux version has the most eyes on it.

My understanding, not having dug into it, is that it's possible but just work nobody has done yet, though I'm not sure what the relevant interfaces are in the Linux kernel.

One thing that makes the interfaces in Linux much messier than the FreeBSD ones is that a lot of the core functionality you might like to leverage in Linux (workqueues, basically anything more complicated than just calling kmalloc, and of course any SIMD save/restore state, to name three examples I've stumbled over recently) are marked EXPORT_SYMBOL_GPL or just entirely not exported in newer releases, so you get to reimplement the wheel for those, whereas on FreeBSD it's trivial to just use their implementations of such things and shim them to the Solaris-ish interfaces the non-platform-specific code expects.

So that makes the Linux-specific code a lot heavier, because upstream is actively hostile.

> SIMD save/restore state

I wish someone would come in and convince the kernel devs that "hey, if you want EXPORT_SYMBOL_GPL to have legal weight in a copyleft sense then you can't just slap it onto interfaces for political reasons"

I don't think they care about it having legal weight, that ship sailed long ago when they started advocating for just slapping SYMBOL_GPL on things out of spite; I think they care about excluding people from using their software.

IMO Linus should stop being half-and-half about it and either mark everything SYMBOL_GPL and see how well that goes or stop this nonsense.

I just don't understand why they're so anti-ZFS. I want my data to survive, please...

My impression is that some of the Linux kernel devs are anti-anything that's not GPL-compatible, of any sort, regardless of the particulars.

Linus himself also made remarks about ZFS at one point that were pretty...hostile. [1] [2]

> The fact is, the whole point of the GPL is that you're being "paid" in terms of tit-for-tat: we give source code to you for free, but we want source code improvements back. If you don't do that but instead say "I think this is _legal_, but I'm not going to help you" you certainly don't get any help from us.

> So things that are outside the kernel tree simply do not matter to us. They get absolutely zero attention. We simply don't care. It's that simple.

> And things that don't do that "give back" have no business talking about us being assholes when we don't care about them.

> See?

Note that there's at least one unfixed Linux kernel bug that was found by OpenZFS users, reproducible without using OpenZFS in any way, reported with a patch, and ignored. [3]

So "not giving back" is a dubious claim.

[1] - https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-s...

[2] - https://www.realworldtech.com/forum/?threadid=189711&curpost...

[3] - https://bugzilla.kernel.org/show_bug.cgi?id=212295

Was ifconfig made to be hostile too? :D

(/s)

That's frankly speaking untrue - we have one off-hand comment by one person from Sun for this, and multiple claims opposite.

Also, ZFS goes with the same licensing interpretation as Linus used with AFS.

We see how well "including people into using your software" works out for pushover licenses almost every week now, FreeBSD itself being the prime example, having lost three major companies one after another from its list of users, sponsors, and developers a couple of years ago. Not that there were many to begin with. What Linus and the gang are doing have been working out great over the long term (much better than any competition, at least).

>having lost three major companies one after another from its list of users, sponsors, and developers a couple of years ago.

Which three?

Why don't you think it has legal weight? Or did you mean something else?

As far as know the point of EXPORT_SYMBOL_GPL was to push back on companies like Nvidia who wanted to exploit loopholes in the GPL. That seems to me like a reasonable objective.

Relevant Torvalds quote: https://yarchive.net/comp/linux/export_symbol_gpl.html

Sure, and that alone isn't an unreasonable premise - as he says, intent matters.

But if you're marking interfaces as GPL-only, or implementing taint detection that means if you use a non-SYMBOL_GPL kernel symbol which calls a GPL-only function it treats the non-SYMBOL_GPL symbol as GPL-only and blocks your linking, it gets a bit out of hand.

Building the kernel with certain kernel options makes modules like OpenZFS or OpenAFS not link because of that taint propagation - because things like the lockdep checker turn uninfringing calls into infringing ones.

Or a little while ago, there was a change which broke building on PPC because a change made a non-SYMBOL_GPL call on POWER into a SYMBOL_GPL one indirectly, and when the original author was contacted, he sent a patch reverting the changed symbol, and GregKH refused to pull it into stable, suggesting distros could carry it if they wanted to. (Of course, he had happily merged a change into -stable earlier that just implemented more aggressive GPL tainting and thereby broke things like the aforementioned...)

The Linux kernel has never supported out of tree modules like how ZFS works out of tree.

All ZFS needs to do is just have one of Oracles many lawyers say "CDDL is compatible with GPL". Yet, they Oracle don't.

"All."

It's explicitly not compatible with GPL, though. It has clauses that are more restrictive than GPL, and IIRC some people who contributed to the OpenZFS project did so explicitly without allowing later CDDL license revisions, which removes Oracle's ability to say CDDL-2 or whatever is GPL-compatible.

So even if someone rolled up dumptrucks of cash and convinced Oracle that everything was great, they don't have all the control needed to do that.

To have legal weight, it has to be a signal that you're implementing something that is derivative of kernel code. That's the directly stated intent of EXPORT_SYMBOL_GPL.

But "call an opaque function that saves SIMD state" is obviously not derivative of the kernel code in any way. The more exports that get badly marked this way, the more EXPORT_SYMBOL_GPL becomes indistinguishable from EXPORT_SYMBOL.

I see it as just a kind of "warranty void if seal broken". Don't do this or you may be in violation of the GPL. Maybe a legal court in $country would find in your favour (I'm not convinced it's as clear cut as you imply). Maybe they would find that you willfully infringed, despite the kernel devs clearly warning you not to do it.

The main "legal effect" I see is that you are not willing to take that risk, just like Oracle isn't.

If the goal was to prove someone saw the GPL, then the entire license enforcement mechanism wouldn't be in the kernel.

If enough symbols get restricted without valid copyright-based reasons, modules that legitimately have non-GPL licenses will have to lie to the kernel to get loaded. And that's a stupid situation all around.

I suspect the underlying issues for not unifying the two have more to do with the ZFS design than anything to do with Linux. It may be the codebase is far too large at this stage to make such a fundamental change.

I don't think so. The memory management stuff is pretty well abstracted; on FBSD it just glues into UMA pretty transparently, it's just on Linux there's a lot of machinery for implementing our own little cache allocating because Linux's kernel cache allocator is very limited in what sizes it will give you, and sometimes ZFS wants 16M (not necessarily contiguous) regions because someone said they wanted 16M records.

The ZoL project lead said at one point there were a variety of reasons this wasn't initially done for the Linux integration [1], but that it was worth taking another look at since that was a decade ago now. Having looked at the Linux memory subsystems recently for various reasons, I would suspect the limiting factor is that almost all the Linux memory management functions that involve details beyond "give me X pages" are SYMBOL_GPL, so I suspect we couldn't access whatever functionality would be needed to do this.

I could be wrong, though, as I wasn't looking at the code for that specific purpose, so I might have missed functionality that would provide this.

[1] - https://github.com/openzfs/zfs/issues/10255#issuecomment-620...

Behlendorf's comment in that thread seems to be talking about linux integration. My point was this is an older issue, going back to the Sun days. See for instance this thread in where McVoy complains about the same issue! https://www.tuhs.org/pipermail/tuhs/2021-February/023013.htm...

That seems more like it's complaining about it not being the actual page cache, not it not being counted as "cache", which is a larger set in at least Linux than just the page cache itself.

But sure, it's certainly an older issue, and given that the ABD rework happened, I wouldn't put anything past being "feasible" if the benefits were great enough.

(Look at the O_DIRECT zvol rework stuff that's pending (I believe not merged) for how a more cut-through memory model could be done, though that has all the tradeoffs you might expect of skipping the abstractions ZFS uses to minimize the ability of applications to poke holes in the abstraction model and violate consistency, I believe...)

Sun has picked the license specifically to make the project incompatible with the Linux kernel. If you start out with hostility, what treatment do you expect in return?

Surely you have a source for that that isn't someone else's comment.

Could the linux integration use dax[0] to bypass the page cache and go straight to ARC?

[0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt

This is amazing. A detail of Netflix that, I a plebe, wouldn’t know if not for this site.

Actually, Drew's presentations about Netflix, FreeBSD, ZFS, saturating high-bandwidth network adapters, etc. are legendary and have been posted far and wide. But having him available to answer questions on HN just takes it to a whole 'nother level.

You're making me blush.. But, to set the record straight: I actually know very little about ZFS, beyond basic user/admin knowledge (from having run it for ~15 years). I've never spoken about it, and other members of the team I work for at Netflix are far more knowledgeable about ZFS, and are the ones who have managed the conversion of our fleet to ZFS for non-content partitions.

Have they ever blogged or spoke at conferences about it? I soak up all that content -- least I try to.

I’ve devoured your FreeBSD networking presentations but I guess I must have confused a post about tracking down a ZFS bug in production written by someone else with all the other content you’ve produced.

Back to the topic at hand, it’s actually scary how few software expose control over whether or not sendfile is used, assuming support is only a matter of OS and kernel version but not taking into account filesystem limitations. I ran into a terrible Samba on FreeBSD bug (shares remotely disconnected and connections reset with moderate levels of concurrent ro access from even a single client) that I ultimately tracked down to sendfile being enabled in the (default?) config - so it wasn’t just the expected “performance requirements not being met” with sendfile on ZFS but even other reliability issues (almost certainly exposing a different underlying bug, tbh). Imagine if Samba didn’t have a tubeable to set/override sendfile support, though.

what do you use for the content filesystem?

FreeBSD's UFS

Any use of boot environments for easy(er?) rollbacks of OS updates?

Yes. That's the bootonce thing I was talking about. When we update the OS, we set the "bootonce" flag via bectl activate -t to ensure we fall back to the previous BE if the current BE is borked and not bootable. This is the same functionality we had by keeping a primary and secondary root partition in geom a and toggling the bootable partition via the bootonce flag in gpart.

If you can share, what type of non-content data do the nodes store? Is this just OS+application+logs?

Running in production for about 3 years with Ubuntu 20.04 / zfs 0.8.3. ZFS is being used as the datastore for a cluster of LXD/LXC instances over multiple physical hosts. I have the OS setup on its own dedicated drive and ZFS striped/cloned over 4 NVMe drives.

No gotchas / issues, works well, easy to setup.

I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018

edit: one thing I forgot to mention is, when creating your pool make sure to import your drives by ID (zpool import -d /dev/disk/by-id/ <poolname>) instead of name in case name assignments change somehow [1]

[1] https://superuser.com/questions/1732532/zfs-disk-drive-lette...

> I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018

See also "Scaling ZFS for NVMe" by Allan Jude at EuroBSDcon 2022:

* https://www.youtube.com/watch?v=v8sl8gj9UnA

Sweet, I appreciate the link!

Sorry if this is a newbie question - but what are you using to share the ZFS file system between physical hosts?

I’ve been out of this particular game for a long time.

In our case each node's ZFS store operates independently, it's not shared. We spin up "VMs" (LXC based) on each node and they talk to each via our private network. Hope that helps!

Sure; we get good mileage out of compression and snapshots (well, mostly send-recv for moving data around rather than snapshots in-place). I think the only problems have been very specific to our install process (non-standard kernel in the live environment; if we used the normal distro install process it would be fine).

I use it as my default file system on FreeBSD. It was rough in FreeBSD 7.x (around 2009), but starting with FreeBSD 8.x it has been rock solid to this day. The only gotcha (which the documentation warns about) has been that automatic block level deduplication is only useful in a few special applications and has a large main memory overhead unless you can accept terrible performance for normal operations (e.g. a bandwidth limited offsite backup).

Yes and it is awesome, no issues.

Yes, Ubuntu 20.04 and 22.04. But we've been running ZFS in some form or other for 10+ years. ACL support not as good/easy to use as Solaris/FreeBSD. Not having weird pathological performance issues with kernel memory allocation like we had with FreeBSD though. Sometimes we have issues with automatic pool import on boot, so that's something to be careful with. The tooling is great though, and we've never had catastrophic failure that was due to ZFS, only due to failing hardware.

Not in production, but using ZoL on my personal workstations. https://zfsonlinux.org/

Some discussion: https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...

The gotcha on Proxmox is that you can't do swapfiles on ZFS, so if your swap isn't made big enough when installing and you format everything as ZFS you have to live with it or do odd workarounds.

That's not a gotcha on Proxmox projects but a gotcha everywhere, SWAP on ZFS needs a lot of special care to avoid that it needs to allocate memory during swapping memory out (e.g., if low on memory), causing hard-freezes.

It can be made somewhat[0] stable with using no (or a simple compression like ZLE), avoiding log devices and caches it can work, but it's way simpler and guaranteed stable to just use a separate partition.

[0] even setting the myriad of options there still exist reports about hangs, like https://github.com/openzfs/zfs/issues/7734#issuecomment-4167...

1. You shouldn't swap on ZFS, but really 2. You shouldn't swap on a filesystem, but really 3. You shouldn't swap at all.

On a VM host you have a bit of leeway for memory overcommit, but ideally if you have a box that frequently has to rely on swap, something is bad wrong. I don't think it's really fair to criticize a worst-choice configuration for being poorly supported here. There are simply too many opportunities for catch-22/deadlock scenarios in the interaction of the fs and allocator during OOM...

Yes. Using latest ZFS ZFS On Linux distrib on Debian. Using Proxmox. Never had and problems ever, ever.

Any of the enterprise customers of klara Systems are likely ZFS production folks.

https://klarasystems.com/?amp

We use ZFS on every server we deploy

We typically use Proxmox. It’s a convenient node host setup and usually has a very up to date zfs and it’s stable

I just wouldn’t use the Proxmox web ui for zfs configuration. It doesn’t have up to date options. Always configure zfs on the cli

> I just wouldn’t use the Proxmox web ui for zfs configuration. It doesn’t have up to date options.

What options are missing for you?

Would be great if you could open an enhancement request over at https://bugzilla.proxmox.com/ for tracking this (no promises on (immediate) implementation though).

TrueNAS (www.truenas.com) uses ZFS for the storage layer across it's product range (storage software).

They have both FreeBSD and Linux based stuff, targeting different use cases.

I use TrueNAS (FreeBSD) version. ZFS makes keeping data snapshots and external backups a fairly easy process. It really cut down on server data management time. The only issue I found with it is there was a now long patched SMB bug a few years ago in the TrueNAS software that caused my storage pool to become corrupted after a large transfer. And there was no way for me to recover the data. It was all lost despite the disks still spinning and presumably still having most and the 1 and 0s recoverable.

Hopefully you had backups?

In production on SmartOS (illumos) servers running applications and VM's, on TrueNAS and plain FreeBSD for various storage and backups, and on a few Linux-based workstations. Using mirrors and raidz2 depending on the needs of the machines.

We've successfully survived numerous disk failures (a broken batch of HDD's giving all kinds of small read errors, an SSD that completely failed and disappeared, etc), and were in most cases able to replace them without a second of downtime (would have been all cases if not for disks placed in hard-to-reach places, now only a few minutes downtime to physically swap the disk).

Snapshots work perfectly as well. Systems are set up to automatically make snapshots using [1], on boot, on a timer, and right before potentially dangerous operations such as package manager commands as well. I've rolled back after botched OS updates without problems; after a reboot the machine was back in it's old state. Also rolled back a live system a few times after a broken package update, restoring the filesystem state without any issues. Easily accessing old versions of a file is an added bonus which has been helpful a few times.

Send/receive is ideal for backups. We are able to send snapshots between machines, even across different OSes, without issues. We've also moved entire pools from one OS to another without problems.

Knowing we have automatic snapshots and external backups configured also allows me to be very liberal with giving root access to inexperienced people to various (non-critical) machines, knowing that if anything breaks it will always be easy to roll back, and encouraging them to learn by experimenting a bit, to the point where we can even diff between snapshots to inspect what changed and learn from that.

Biggest gotchas so far have been on my personal Arch Linux setup, where the out-of-tree nature of ZFS has caused some issues like a incompatible kernel being installed, the ZFS module failing to compile, and my workstation subsequently being unable to boot. But even that was solved by my entire system running on ZFS: a single rollback from my bootloader [2] and all was back the way it was before.

Having good tooling set up definitely helped a lot. My monkey brain has the tendency to think "surely I got it right this time, so no need to make a snapshot before trying out X!", especially when experimenting on my own workstation. Automating snapshots using a systemd timer and hooks added to my package manager saved me a number of times.

[1]: https://github.com/psy0rz/zfs_autobackup [2]: https://zfsbootmenu.org/

> Systems are set up to automatically make snapshots

I do that with sqlite to keep a selection of snapshots from the last hours, days etc.

https://github.com/csdvrx/zfs-autosnapshot

I use ZFS on Debian for my home file server. The setup is just a tiny NUC with a couple of large USB hard drives, mirrored with ZFS. I’ve had drives fail and painlessly replaced and resilvered them. This is easily the most hassle free file storage setup I’ve owned; been going strong over 10 years now with little to no maintenance.

Pretty much same setup here except I used a Lacie atom NAS, added a USB stick to the motherboard for booting.

I have been using zfs for years from ubuntu 18. Easy snapshots, monitoring, choices for raid levels, and ability to very easily copy a dataset remotely with resume or incremental support is awesome. I mainly use it for kvm systems each with their own dataset. Coming from mdadm + lvm from my previous set up is night and day for doing snapshots and backups. I do not use zfs on root for ubuntu instead I do a raid1 software set up for the os and then a zfs set up on other disks - zfs on root was the only gotcha. For FreeBSD zfs on root works fine.

ZFS on Linux has improved a lot in the last few years. We (prematurely) moved to using it in production for our MySQL data about 5 years ago and initially it was a nightmare due to unexplained stalling which would hang MySQL for 15-30 minutes at random times. I'm sure it shortened my life a few years trying to figure out what was wrong when everything was on fire. Fortunately, they have resolved those issues in the subsequent releases and it's been much more pleasant after that.

"production" at home on Debian 11, previously on FreeBSD 10-13. The weirdest gotcha has been related to sending encrypted raw snapshots to remote machines[0],[1]. These have been the first instabilities I had with ZFS in roughly 15 years around the filesystem since switching to native encryption this year. Native encryption seems to be barely stable for production use; no actual data corruption but automatic synchronization (I use znapzend) was breaking frequently. Recent kernel updates fixed my problem although some of the bug reports are still open. I only moved on from FreeBSD because of more familiarity with Linux.

A slightly annoying property of snapshots and clones is the inability to fully re-root a tree of snapshots, e.g. permanently split a clone from its original source and allow first-class send/receive from that clone. The snapshot which originated the clone needs to stick around forever[2]. This prevents a typical virtual machine imagine process of keeping a base image up to date over time that VMs can be cloned from when desired and eventually removing the storage used by the original base image after e.g. several OS upgrades.

I don't have any big performance requirements and most file storage is throughput based on spinning disks which can easily saturate the gigabit network.

I also use ZFS on my laptop's SSD under Ubuntu with about 1GB/s performance and no shortage of IOPS and the ability to send snapshots off to the backup system which is pretty nice. Ubuntu is going backwards on support for ZFS and native encryption uses a hacky intermediate key under LUKS, but it works.

[0] https://github.com/openzfs/zfs/issues/12014 [1] https://github.com/openzfs/zfs/issues/12594 [2]https://serverfault.com/questions/265779/split-a-zfs-clone

We don't use ZFS as a filesystem, we use it to aggregate multiple EBS into a single pool, and present it to... stock Ubuntu (one found in AWS marketplace) as a single big block device.

We also use ZFS for testing of our own block device. My biggest problems with it are related to this activity. Hot-plugging or removing block devices from the pool often leads to unrepeatable pool, and the tests have to be scrambled because the whole system needs to be rebooted.

>We don't use ZFS as a filesystem, we use it to aggregate multiple EBS into a single pool, and present it to... stock Ubuntu (one found in AWS marketplace) as a single big block device.

I don't understand what you mean by this comment. Are you creating a zvol on your zpool and loopback mounting it with a different filesystem? It sounds like you are very much using ZFS as a filesystem, even if you are not booting from it or using it for any kind of high availability purpose.

I do agree with you on the matter that there should be a way to clear out an (UNAVAIL vdev / SUSPENDED pool) without a reboot. This has been a thorn in my side for as long as ZFS has existed. The deadlock-by-design line is really quite nonsense.

No, we don't use it as a filesystem at all. Our product is similar to EBS. So, nothing is mounted anywhere. Users of our product will get a slice or a whole volume thus created and usually would put some filesystem on it, but they will see that volume as a LUN in iSCSI portal we give them an address of. So, in principle, they could also use it to create a ZFS filesystem, but I doubt anyone uses it like that. Most typical usages are either filesystem (something like Ext4 or XFS) or an object store.

I use it in Kubernetes via https://github.com/openebs/zfs-localpv

The PersistentVolume API is a nice way to divvy up a shared resource across different teams, and using ZFS for that gives us the snapshotting, deduplication, and compression for free. For our workloads, it benchmarked faster than XFS so it was a no-brainer.

In production since the first Release of FreeBSD in our Research company. We layered our storage to disconnect hardware Lifecycles from OS. The spinning disks are pooled together using ScaleIO and provided to FreeBSD on ESX as physical devices which live forever through each EOL cycle of the hardware every five up to eight years. We have been serving ten PetaBytes for 15 years without hiccups. We expose SMB3/NFS4 and S3 and have nightly DR to mirror sites using ZFS binary difference Send and Receive cronjobs which are fully automated and monitored. Basically the ScaleIO SDS servers can be replaced during daytime while the upper layer keeps going on for HPC and thousands of other jobs. Since hardware provides more capacity after each Lifecycle we can increase the ZFS pool on the fly and have the most profound 24/7 storage environment without any major interruptions for 15 YEARS and counting. Our partners are having multiple outages due Lustre and other problems, my 2c

We use it with SmartOS on hypervisors running customer Bhyve machines. ZFS (and smartos) works really really well. .

The fact how I can replicate live data of arbitrary size (many TB sized filesystems) in small chunks to other hosts every minute greatly increased my deep sleep quality. Of course databases with multi-host write etc are nice, but in our use case, all customers are rather small with just lots and lots of files (medical and otherwise), the database itself is rather small and doesn't need replication.

Best thing, on the receiver side of the backup ZFS ensures due to its architecture that the diff is directly applied on top of the existing filesystem, while in normal differential backups one might find out months or years later that one diff snapshot was damaged in transfer or is not accessible.

zfs scrubbing with S.M.A.R.T monitoring also helps a lot to ensure drive quality over time.

# Gotchas

ZFS:

- There is no undo etc, this is unix, so beware of wrong commands. - ZFS Scrubbing can be stopped (it does sometimes affect io speed), but ZFS resilvering cannot. This can lead to performance issues. - There must be enough RAM for the caching to work well and synchronous workloads do well with good write cache drives (ZIL) - Data usage patterns should fit well with the Append Log schema of ZFS. E.g databases such as LevelDB worked really well. Others are not slow, but need a good ZIL more then when the pattern fits.

SmartOS: Some minor gotchas with how vmadm deletes zfs filesystems or in general with SmartOS, e.g when having too many snapshots, but everything quite predictable.

Yeah, a few years ago I steered our university research group towards using ZFS for a couple of new servers with 1.5PB storage each (2.0PB before raidz2), on RHEL (would have preferred Debian). Absolutely no problems with the system. The IT guys were a bit doubtful whether it would work well, for instance they insisted that XFS would be better at managing parallel requests, but when the system happily saturates a 10Gb ethernet with plenty of bandwidth to spare I don't think there's a problem. We have had ZFS flag up one hard drive so far that was returning false data, which was then hot-swapped out. Other filesystems would probably have just suffered data corruption without us knowing it.

My workplace has been using ZFS for years. I've yet to see an alternative come close to being compelling. We use OpenZFS on all our RHEL servers and even workstations with XFS for root partitions and smaller clients. Our primary file server is Solaris 11 but is coming to the end of its long life. The plan is to use FreeBSD on the replacement. ZFS has become so valuable to us that it alone can be a major factor in directing our other choices.

Is nice to see it advancing in useful ways. Hopefully this leads to offline dedup without the runtime memory costs.