News for hackers. Stuff that matters.
Re:OpenZFS 2.0 by anderspitman on Monday November 30, 2020 @05:18PM (#25259761)
I'd love to get rid of my FreeNAS VM and run ZFS directly on my Linux desktop, but having to mess with the kernel has kept me from attempting it so far. Maybe I'm worrying about nothing.

btrfs seems like the main alternative if you want native kernel support, but when I checked a couple years ago there seemed to be a lot of concerns about the stability. Is that still the case?

Re:OpenZFS 2.0 by briskstorm on Monday November 30, 2020 @06:13PM (#25260266)
To mirror other comments, btrfs is pretty stable now and I run it on my server. The main problems now are with RAID5/6 profiles, the implementation has lots of issues still that can cause data loss[1]. Seems most of the core developers don't use those profiles so it hasn't been getting better. If you want to use RAID it would be safer to stick to RAID1/10[2]

[1] https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@h...

[2] https://www.man7.org/linux/man-pages/man8/mkfs.btrfs.8.html#...

Re:OpenZFS 2.0 by curt15 on Monday November 30, 2020 @06:25PM (#25260372)
On the btrfs mailing list [1], there are still sporadic reports of unrecoverable FS corruption for whatever reason. See [2], [3] for some recent examples.

[1] https://lore.kernel.org/linux-btrfs/

[2] https://lore.kernel.org/linux-btrfs/CAD7Y51i=mTDnEWEJtSnUsq=...

[3] https://lore.kernel.org/linux-btrfs/CAMXR++KUj2L7qpR7QZeiM2T...

Re:OpenZFS 2.0 by viraptor on Tuesday December 01, 2020 @04:02AM (#25263672)
[3] is interesting, thanks for linking that. [2] has so many moving parts, I wouldn't expect it to be related to btrfs without more information. I mean, there's both the fs and the cache layer being resized down, with unknown method.
Re:OpenZFS 2.0 by ariabuckles on Monday November 30, 2020 @05:28PM (#25259859)
Both openSUSE and [as of very recently] Fedora use btrfs by default, so btrfs support seems pretty stable these days.

(But as others have pointed out, there are options for using zfs on linux, too)

Re:OpenZFS 2.0 by bmn__ on Tuesday December 01, 2020 @04:52AM (#25263875)
Attempting to use zfs for the root partition is a huge headache because the software lives in the supplementary `filesystems` repo. https://build.opensuse.org/package/show/filesystems/zfs

1. It often happens that the main repo offers a new kernel, but the corresponding module is not ready on obs yet. This means upgrading to the latest rolling release cannot just happen at any time, but requires careful planning. This is a big inconvenience.

2. In the past dracut sometimes just failed to pick up the module for the initrd, causing a boot failure at the next system start. I could not figure out why, however this never happened with the first class supported ext/xfs.

3. The distro's boot/rescue media do not contain the driver. This means a third-party boot medium is required to go into a broken system, and repairing it when chroot is involved is now much more complicated because of the different distro.

Re:OpenZFS 2.0 by ed25519FUUU on Monday November 30, 2020 @07:46PM (#25260939)
btrfs was a really underutilized filesystem. It still has some superior features to zfs (such as offline deduplication), but the momentum now is clearly with zfs.
Re:OpenZFS 2.0 by Ericson2314 on Monday November 30, 2020 @05:33PM (#25259915)
ZFS is no extra work with NixOS! You just declare the filesystem type like any other in the config and it takes care of kernel modules and what-not.
Re:OpenZFS 2.0 by pbronez on Monday November 30, 2020 @06:22PM (#25260351)
Sure - after you figure out NixOS lol
Re:OpenZFS 2.0 by takeda on Monday November 30, 2020 @07:29PM (#25260845)
Actually no, NixOS is probably easier to use than other Linuxes. It gets more difficult when you need to package something new that it doesn't have, then you have to know the Nix language and how nixpkgs work.
Re:OpenZFS 2.0 by mook on Tuesday December 01, 2020 @02:19AM (#25263149)
Interesting; when I tried it a while back it seemed like you needed to know the language to manage your configurations. Was that impression incorrect?

I may have tried it far enough back that I pretty much immediately encountered packages I wished it had and tried (and failed) to package it myself, though, and got the experience mixed up…

Re:OpenZFS 2.0 by aseipp on Tuesday December 01, 2020 @07:48AM (#25264665)
No, you're 100% correct. Despite the prosthelytization you'll get from many died-in-the-wool users who forgot what it was like in the beginning, in order to use NixOS effectively you either need to learn Nix, or be willing to spend a lot of time on IRC asking questions, which will end up with... you learning Nix. That's the reality. I think I spent something like 3 weeks porting my server configuration from Ubuntu to NixOS, by hand, piece by piece, many years ago. Admittedly I think we're better off than we were 7 years ago, but it's still not a grand slam. Even things like basic GUI installers that can set up your filesystem don't exist! Manually screwing with partition layouts to get volume encryption isn't easy to use at all, honestly.

There are an array of flaws with the tools but, despite that, they are unbelievable powerful and you can do things in NixOS you can't dream of doing elsewhere, and it makes things like using OpenZFS or whatever pretty easy and simple. And it makes some thing far more difficult than that, nearly trivial. But only once you know what you're doing. But that's just the reality: it's an extremely powerful tool that has many rough edges. Saying it's "the easiest distro to use" is a complete joke, and I wish fellow NixOS users didn't have some weird propensity to practically lie about how good it is on that front. I say this as someone who has been a NixOS developer and user for like, ~7 years and who apparently(?!) has over 1,000 commits to the tree now, too. Trying to actually sit down at a terminal and sell unconvinced people on it opened my eyes quite a bit. It's good, but lying about what it is and isn't is a good way to burn peoples faith.

Re:OpenZFS 2.0 by takeda on Wednesday December 02, 2020 @01:13AM (#25273718)
I don't think what you said necessarily contradicts what I said.

NixOS is easy to use as long as as what you're trying to do is contained within the configuration, then set up is pretty much just editing that configuration file (which essentially is just series of dictionaries and lists). I wouldn't for example have with installing NixOS for my grandparents.

Here's example config when that's true: https://github.com/areina/nixos-config/blob/master/thinkpad-...

If you want to do something that's not covered then you'll have to learn Nix, and that part is indeed hard, because it's like configuring linux through use of saltstack/chef/puppet/ansible through a functional language (which many people don't have experience with), but as you said it pays off.

I think the hardest part is the paradigm shift where everything you do is no longer imperative but declarative. It also doesn't help that documentation is always behind what nixpkgs can do and Nix functionality would cover multiple books.

Re:OpenZFS 2.0 by brnt on Monday November 30, 2020 @06:06PM (#25260193)
Is there a guide for this? Sounds interesting!
Re:OpenZFS 2.0 by jfb on Monday November 30, 2020 @06:21PM (#25260342)
For instance, I have:

     fileSystems."/zfs/media" =
        { device = "tank/media";
          fsType = "zfs";
        };
in my hardware-configuration.nix. tank/media is defined as using a legacy mount-point or whatever the ZFS terminology is. Done.

ETA: I mean, I had to do all the gruntwork to get the pool built, yeah. But once it was defined, getting it mounted and all the kernel bits and bobs set was trivial like that.

Re:OpenZFS 2.0 by evil-olive on Monday November 30, 2020 @08:16PM (#25261130)
In addition, if you just want to play around with it:

    boot.supportedFilesystems = [ "zfs" ];
Both installs the necessary kernel modules and adds zpool(1) / zfs(1) to $PATH.
Re:OpenZFS 2.0 by linsomniac on Monday November 30, 2020 @07:00PM (#25260640)
I've been running encrypted ZFS on 20.04 on my main workstation since it came out and it's worked great. Wrote up details here, it's a slight hack for encryption, no hack if you don't want crypto. https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc...

A friend did a video based on my blog: https://www.youtube.com/watch?v=PILrUcXYwmc

Re:OpenZFS 2.0 by koolba on Monday November 30, 2020 @10:01PM (#25261807)
Why ZFS encryption vs unencrypted ZFS atop LUKS?
Re:OpenZFS 2.0 by necheffa on Monday November 30, 2020 @10:39PM (#25262036)
Native ZFS encryption make pool management easier. Less bookkeeping to import/export a pool. And the file system can make better decisions about performance and recovery.
Re:OpenZFS 2.0 by phil21 on Monday November 30, 2020 @10:30PM (#25261983)
I wanted to say "as someone who tends to follow the unix philosophy" and realized the irony of saying that regarding ZFS...

That said, I generally agree with you in that do one thing and do it well is a laudable design goal. However, I also am very excited about encrypted ZFS for one main reason: backups.

Okay two. Snapshots and backups!

ZFS is absolutely amazing to use as a home NAS that does daily (or more) snapshots and then nightly differential syncs to a second location. In the past I had to run all my own infrastructure to do this, as the data was in the clear.

Now my ZFS nerd friend and I can simply swap backup space and have "zero knowledge" of the others' files, while retaining the amazing features of ZFS snapshots+zfs send/receive.

This also tickles the "create an encrypted ZFS backups as a service" service itch for me, but then I realize I'd be creating it for all 13 potential users of the service. That said, I'm sure rsync.net will offer this functionality shortly - which would make them a viable backup target for me.

Re:OpenZFS 2.0 by p_l on Tuesday December 01, 2020 @12:45AM (#25262719)
Regarding "unix philosophy" and ZFS... it's actually very compliant with it, despite uninformed claims from early public release about "blatant layering violations".

It's just that majority of users never have reason to see more than tiny signs of the layers hidden behind (mostly) 2 command line tools, and for various reasons those layers are compiled into one one module.

But the clean layered design is how LustreZFS happened :)

Re:OpenZFS 2.0 by stubish on Tuesday December 01, 2020 @01:11AM (#25262856)
https://zfs.rent/ is very interesting.

I also recall someone working years ago on a way to push snapshots to S3 or similar, but I never heard if that idea got off the ground (downside is of course the snapshots need to be recovered before they can be mounted, but the dollar cost would be rock bottom).

What would be more interesting is a backup application for Desktop Linux that assumes a ZFS root; all the problems that plague Desktop applications (that seem to keep them in eternal beta or wither away) disappear. It needs to switch on and push snapshots. It needs allow the remote file system to be mounted (to browse the snapshots for selective recovery). It needs a a disaster recovery process to recover an entire system from a remote snapshot.

Re:OpenZFS 2.0 by hrez on Wednesday December 02, 2020 @10:18PM (#25284929)
> I also recall someone working years ago on a way to push snapshots to S3

You can pipe zfs send to gof3r.

Re:OpenZFS 2.0 by rakoo on Tuesday December 01, 2020 @05:34AM (#25264065)
rsync.net already provides zfs receive capability: https://www.rsync.net/products/zfsintro.html
Re:OpenZFS 2.0 by brnt on Tuesday December 01, 2020 @07:43AM (#25264643)
Once you put a fs on top of something, don't you lose any guarantees of finding broken sectors when scrub?

This is why I really wish btrfs would get native encryption, but maybe my info is out of date.

Re:OpenZFS 2.0 by yjftsjthsd-h on Monday November 30, 2020 @10:10PM (#25261866)
Why add another separate layer?
Re:OpenZFS 2.0 by linsomniac on Monday December 07, 2020 @01:30PM (#25335328)
Sorry for the late reply, traveling.

In my case, because it's what Ubuntu "supports" for bootable root crypto ZFS, and I wanted to try it.

I've run ZFS on top of LUKS for my backup storage servers for probably over a decade now, and it works fine. But it wasn't really an option for my workstation.

That said, I'm not really sure what benefit I'm gaining from ZFS on my desktop. I've got that snapshots which are definitely nice. I've used it a couple times to go back in time. In theory I can go back if I wedge the system through package installs or an OS upgrade, but I've not done that (yet). It does slow down package installs because of taking snapshots, but that's ok generally.

Re:OpenZFS 2.0 by koolba on Monday November 30, 2020 @10:13PM (#25261882)
Because traditionally, programs do one thing well.

And in my experience LUKS works great.

Re:OpenZFS 2.0 by yjftsjthsd-h on Monday November 30, 2020 @10:49PM (#25262095)
If you're using ZFS, you're already okay with some level of ignoring that; ZFS is a inherently huge layering violation. It's a filesystem and a volume manager with encryption and compression, its own user access system (zfs allow), and its own NFS implementation.
Re:OpenZFS 2.0 by p_l on Tuesday December 01, 2020 @12:48AM (#25262734)
Umm, that's untrue.

It's a very cleanly layered system, it just doesn't bother end user with details (as implementor, you can play with them, thus LustreZFS): there's separate SPA (block), DMU (OSD) and ZPL (FS) & ZVOL (emulated block device) layers.

Compression and encryption are integrated at DMU level because that's a logical place for them.

NFS actually calls OS nfs server.

Re:OpenZFS 2.0 by yjftsjthsd-h on Tuesday December 01, 2020 @02:11AM (#25263116)
I suppose it depends which layer you look at; as a humble sysadmin/user, it certainly behaves like a monolithic system, as opposed to having separate things to configure and invoke for LVM/LUKS/filesystem (which is, after all, the starting topic for this thread). Having only semi-recently seen an internal architecture diagram, I was indeed pleased to see that it's internally a composition of parts that do one thing well, but that's not really user-visible.

Interesting; it'd been claimed to me before that ZFS had its own NFS server (or I guess the OpenSolaris NFS server) included but that nobody used it because it was old/buggy. A quick glance at https://github.com/openzfs/openzfs/ (the old archived version based on illumos, if I read correctly) implies that this might have been true at one point, but indeed https://github.com/openzfs/zfs doesn't seem to do its own NFS so it's not true now if it ever was. Thanks for correcting my understanding.

Re:OpenZFS 2.0 by mook on Tuesday December 01, 2020 @02:15AM (#25263136)
That sounds about as modular as systemd: it's in theory modular, but it very much throws away the existing boundaries and has yet to see its new modules be adopted by external folks (if that were even possible, depending on interface stability).

I believe that the modularity will only proof itself when external (as in, from unrelated people) projects becomes established and we see how well the original project maintains compatibility.

(I must say I wonder how big the intersection of people-who-like-ZFS and people-who-like-systemd is; they seemed to originate from very different cliques but there's no reason people who like one would dislike the other…)

Re:OpenZFS 2.0 by p_l on Tuesday December 01, 2020 @09:52AM (#25265646)
There's sufficiently strong dislike for systemd in the ZoL circles I moved around, because ZFS was all about making sure its rock solid and stable, while systemd has well-deserved reputation for breaking things often.

As for external clients for the layers, LustreZFS is a separate project though it had started with certain intersection of ZFS devs. However the general division of labour between layer is pretty strict (except for - now extinct - FreeBSD TRIM support), it's just that there isn't any work being done to use it outside of OpenZFS.

The boundaries are pretty clear, it's just that ZPL and ZVOL build up on all of them. Linux has /some/ related features, but nothing that was feature-parity: SPA roughly corresponds with MD/DM subsystem assuming certain plugins in use, DMU is very roughly the equivalent of OSD subsystem, but that one supports only SCSI OSD which has incompatible assumptions etc. - in fact, an OSD implementation on top of DMU should be pretty simple (main differences are due to DMU being a bit explicit on redundancy features, iirc).

Re:OpenZFS 2.0 by johnisgood on Tuesday December 01, 2020 @11:09PM (#25273146)
Does it offer plausible deniability like dm-crypt?
Re:OpenZFS 2.0 by kbumsik on Monday November 30, 2020 @05:39PM (#25259956)
I personally use ZFS on Arch Linux. The DKMS package works almost out of the box and I haven't had any troubles. It takes a long time (but not too much) to compile though.

Or you can use the latest Ubuntu that is shipped with ZFS.

Re:OpenZFS 2.0 by nvllsvm on Monday November 30, 2020 @06:14PM (#25260278)
Be aware of possible incompatibilities with the regular Arch kernel. I switched my NAS to linux-lts when a kernel point-release proved incompatible with openzfs.
Re:OpenZFS 2.0 by kbumsik on Monday November 30, 2020 @09:11PM (#25261512)
Right, I forgot to mention I always use the LTS kernel, since I had getrandom() booting problem last year. https://lwn.net/Articles/800509/
Re:OpenZFS 2.0 by ed25519FUUU on Monday November 30, 2020 @07:49PM (#25260953)
Reporting in from Ubuntu 20.04 with ZFS on root. It works great. No issues so far, other than docker requires "zfs" as the storage-driver and doesn't support overlay2. See: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/171...
Re:OpenZFS 2.0 by kbumsik on Monday November 30, 2020 @09:13PM (#25261521)
Interesting. I thought overlay2 can be agnostic to the base filesystems.
Re:OpenZFS 2.0 by heavyset_go on Monday November 30, 2020 @07:54PM (#25260996)
I use btrfs on both servers and laptops without a problem these days. This wasn't the case almost a decade ago, though, when I got bit by its then-instability.
Re:OpenZFS 2.0 by paulsmal on Monday November 30, 2020 @05:21PM (#25259787)
You know Ubuntu support ZFS since 20.04. Experimental, but quite stable for me. Just select file system during installation process.
Re:OpenZFS 2.0 by sliken on Monday November 30, 2020 @05:24PM (#25259824)
Or say apt install zfs-utils, that's it, not even a reboot.
Re:OpenZFS 2.0 by paulsmal on Monday November 30, 2020 @05:37PM (#25259937)
Oh yeah, right. Installation method if one needs ZFS on root.
Re:OpenZFS 2.0 by rodgerd on Monday November 30, 2020 @06:50PM (#25260570)
> but having to mess with the kernel has kept me from attempting it so far. Maybe I'm worrying about nothing.

For the most part, yes. Occasionally a kernel developer who seems to be bitter about a company that doesn't exist any more tries to break compat with ZFS, but it's generally smooth sailing on Fedora, Debian, and CentOS, with dkms handling the building of modules seamlessly.

Re:OpenZFS 2.0 by plmu on Tuesday December 01, 2020 @02:42AM (#25263257)
Just use something else for your root file system, and zfs for the rest. I've been running Zol for 10 years (on arch) and had to recover a few times, but it was never difficult because of the totally standard setup except for the data disks.
Re:OpenZFS 2.0 by weitzj on Monday November 30, 2020 @05:29PM (#25259873)
The easiest way is using t Proxmox installer which has ZFS as a filesystem. Underneath it is a Debian 10 installation. Last time I tried you could not enable ZFS encryption. I don’t know what is the case with Openzfs 2.0

Do we have encryption,yet?

Re:OpenZFS 2.0 by vetinari on Monday November 30, 2020 @06:09PM (#25260227)
Proxmox uses their own kernels, so they build the zfs modules as well. They are bundled in the pve-kernel package.
Re:OpenZFS 2.0 by paulsmal on Monday November 30, 2020 @05:38PM (#25259953)
Ubuntu 20.10 has an option in installer to use ZFS encryption for root partition.
Re:OpenZFS 2.0 by remram on Monday November 30, 2020 @09:35PM (#25261652)
Oh sweet, I don't think encryption was an option in 20.04. Is it still under an "experimental" option in the installer?
Re:OpenZFS 2.0 by paulsmal on Tuesday December 01, 2020 @03:16AM (#25263429)
Yeah, still experimental but improved over 20.04. Added encryption and "autotrim=on" by default.

I think Canonical planing is to have ZFS as an experimental option only for Desktop version of Ubuntu until next LTS release.

Re:OpenZFS 2.0 by Jnr on Monday November 30, 2020 @05:48PM (#25260044)
I'm using btrfs and my system still works. :)
Re:OpenZFS 2.0 by muxator on Monday November 30, 2020 @08:37PM (#25261282)
Me too. I have been running full btrfs since at least 5 years (single and RAID1 disks), which no data loss whatsoever. Now that there is support for swapfiles, I do not even need to create a dedicated partition for them.
Re:OpenZFS 2.0 by nix23 on Monday November 30, 2020 @05:26PM (#25259847)
>ZFS directly on my Linux desktop

Use BTRFS trust me it's stable now...well the commands are terrible compared to ZFS. All my Server are FreeBSD but on the Laptop and on one Workstation i have openSUSE Tumbleweed since like 2 years and it works great.

Re:OpenZFS 2.0 by raegis on Monday November 30, 2020 @07:41PM (#25260901)
I've used btrfs for years with no problems ever. But, I see weekly reports on the btrfs reddit forum of the type "I was doing btrfs RAIDxyz, and I can't mount read/write" etc., so there do exist people who have issues with it today. If you do RAID on steroids, you might do some research before trying btrfs.
Re:OpenZFS 2.0 by gpanders on Monday November 30, 2020 @05:47PM (#25260039)
> well the commands are terrible compared to ZFS

Really? I don’t think so, I find btrfs usage extremely straightforward and easy to grok. ZFS on the other hand has all that confusing lingo about vdevs, etc...

I get that this is subjective but I disagree.

Re:OpenZFS 2.0 by rleigh on Tuesday December 01, 2020 @04:24AM (#25263757)
The Brtfs commands are very poor compared with what ZFS offers. The ZFS commands are organised around the end user: the system administrator. The Brtfs commands are not.

As an example, you're running low on space and need to find out which datasets (subvolumes) are using the most space. How do you do that? With ZFS it's a single command which runs in a few milliseconds. With Btrfs...

Re:OpenZFS 2.0 by nix23 on Monday November 30, 2020 @06:04PM (#25260173)
Hey everyone has a different taste, but vdevs, datasets, and pool are for me much more logical than lv's and lg's (pun was NOT intended).
Re:OpenZFS 2.0 by zeropoint46 on Monday November 30, 2020 @06:44PM (#25260530)
but thats not really btrfs, thats LVM. I use BTRFS directly on physical disks and dont use pvs, vgs or lvs.
Re:OpenZFS 2.0 by nix23 on Monday November 30, 2020 @08:31PM (#25261229)
well then call it volumes and vdev's?...i love zfs's layering.
Re:OpenZFS 2.0 by neolog on Monday November 30, 2020 @05:47PM (#25260032)
> the commands are terrible

what does that mean?

Re:OpenZFS 2.0 by chungy on Tuesday December 01, 2020 @02:46AM (#25263285)
The "btrfs" tool has a lot of leaky abstractions, confusing intended usage, and gotchas all over the place. If you aren't a btrfs developer, it is difficult to know what exactly you want to do and how to accomplish it.

ZFS on the other hand has just two commands for common administration tasks: zpool and zfs. zpool controls pool-level operations, mainly ones that deal with the storage layer; zfs controls the logical file systems and volumes that are contained within a pool. The zpool and zfs commands have been meticulously crafted to not expose much of the underlying software architecture and focus only on what administrators want, and all of it is clearly documented.

There are actually a few other commands that come with ZFS if you really want or need to deal with low-level and difficult details, commands like zdb, zinject, zstreamdump. You almost never need any of them.

Re:OpenZFS 2.0 by vetinari on Monday November 30, 2020 @06:05PM (#25260179)
For zfs specific features, there are 'zfs' and 'zpool' commands (well, binaries, and the first parameter is a command). For btrfs, there is 'btrfs'.

So I guess that the GP considers /usr/sbin/{zfs,zpool} more intuitive than /usr/sbin/btrfs.

Re:OpenZFS 2.0 by nix23 on Monday November 30, 2020 @06:13PM (#25260263)
Well zpool is for the pool which consist of filesystems/datasets or zfs's. So i have a clear distinction of the whole pool (pools or logical groups) or filesystems/datasets (zfs or logical volumes).

It has nothing todo with /usr/sbin/x

Re:OpenZFS 2.0 by nix23 on Monday November 30, 2020 @06:07PM (#25260208)
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs

>what does that mean?

Not functional but logical (for me)

Re:OpenZFS 2.0 by arwineap on Monday November 30, 2020 @05:24PM (#25259825)
debian has zfs in the contrib repo since stretch; no manual hacking required just have to enable contrib

I switched my freebsd box over to debian about two years ago. No complaints so far :)