News for hackers. Stuff that matters.
Re:ZFS 2.2.0 (RC): Block Cloning merged by drewg123 on Tuesday July 04, 2023 @01:03PM (#36589424)
We use ZFS in production for the non-content filesystems on a large, ever increasing, percentage of our Netflix Open Connect CDN nodes, replacing geom's gmirror. We had one gotcha (caught on a very limited set of canaries) where a buggy bootloader crashed part-way through boot, leaving the ZFS "bootonce" stuff in a funky state requiring manual recovery (the nodes with gmirror were fine, and fell back to the old image without fuss). This has since been fixed.

Note that we do not use ZFS for content, since it is incompatible with efficient use of sendfile (both because there is no async handler for ZFS, so no async sendfile, and because the ARC is not integrated with the page cache, so content would require an extra memory copy to be served).

Re:ZFS 2.2.0 (RC): Block Cloning merged by bakul on Tuesday July 04, 2023 @01:33PM (#36589913)
Is integrating ARC with the page cache a lost cause? If not, may be Netflix can fund it!
Re:ZFS 2.2.0 (RC): Block Cloning merged by postmodest on Tuesday July 04, 2023 @02:10PM (#36590475)
I would expect page cache to be one of those platform-dependent things that prevent OpenZFS from doing that. Especially on Linux, and especially because AFAIK the Linux version has the most eyes on it.
Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @03:27PM (#36591491)
My understanding, not having dug into it, is that it's possible but just work nobody has done yet, though I'm not sure what the relevant interfaces are in the Linux kernel.

One thing that makes the interfaces in Linux much messier than the FreeBSD ones is that a lot of the core functionality you might like to leverage in Linux (workqueues, basically anything more complicated than just calling kmalloc, and of course any SIMD save/restore state, to name three examples I've stumbled over recently) are marked EXPORT_SYMBOL_GPL or just entirely not exported in newer releases, so you get to reimplement the wheel for those, whereas on FreeBSD it's trivial to just use their implementations of such things and shim them to the Solaris-ish interfaces the non-platform-specific code expects.

So that makes the Linux-specific code a lot heavier, because upstream is actively hostile.

Re:ZFS 2.2.0 (RC): Block Cloning merged by Dylan16807 on Tuesday July 04, 2023 @04:40PM (#36592266)
> SIMD save/restore state

I wish someone would come in and convince the kernel devs that "hey, if you want EXPORT_SYMBOL_GPL to have legal weight in a copyleft sense then you can't just slap it onto interfaces for political reasons"

Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @05:01PM (#36592459)
I don't think they care about it having legal weight, that ship sailed long ago when they started advocating for just slapping SYMBOL_GPL on things out of spite; I think they care about excluding people from using their software.

IMO Linus should stop being half-and-half about it and either mark everything SYMBOL_GPL and see how well that goes or stop this nonsense.

Re:ZFS 2.2.0 (RC): Block Cloning merged by Filligree on Tuesday July 04, 2023 @05:15PM (#36592580)
I just don't understand why they're so anti-ZFS. I want my data to survive, please...
Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @05:25PM (#36592693)
My impression is that some of the Linux kernel devs are anti-anything that's not GPL-compatible, of any sort, regardless of the particulars.

Linus himself also made remarks about ZFS at one point that were pretty...hostile. [1] [2]

> The fact is, the whole point of the GPL is that you're being "paid" in terms of tit-for-tat: we give source code to you for free, but we want source code improvements back. If you don't do that but instead say "I think this is _legal_, but I'm not going to help you" you certainly don't get any help from us.

> So things that are outside the kernel tree simply do not matter to us. They get absolutely zero attention. We simply don't care. It's that simple.

> And things that don't do that "give back" have no business talking about us being assholes when we don't care about them.

> See?

Note that there's at least one unfixed Linux kernel bug that was found by OpenZFS users, reproducible without using OpenZFS in any way, reported with a patch, and ignored. [3]

So "not giving back" is a dubious claim.

[1] - https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-s...

[2] - https://www.realworldtech.com/forum/?threadid=189711&curpost...

[3] - https://bugzilla.kernel.org/show_bug.cgi?id=212295

[dead]
Re:ZFS 2.2.0 (RC): Block Cloning merged by olgeni on Wednesday July 05, 2023 @04:39AM (#36597537)
Was ifconfig made to be hostile too? :D

(/s)

Re:ZFS 2.2.0 (RC): Block Cloning merged by p_l on Monday July 10, 2023 @04:18AM (#36663459)
That's frankly speaking untrue - we have one off-hand comment by one person from Sun for this, and multiple claims opposite.

Also, ZFS goes with the same licensing interpretation as Linus used with AFS.

Re:ZFS 2.2.0 (RC): Block Cloning merged by 5e92cb50239222b on Wednesday July 05, 2023 @03:11AM (#36596670)
We see how well "including people into using your software" works out for pushover licenses almost every week now, FreeBSD itself being the prime example, having lost three major companies one after another from its list of users, sponsors, and developers a couple of years ago. Not that there were many to begin with. What Linus and the gang are doing have been working out great over the long term (much better than any competition, at least).
Re:ZFS 2.2.0 (RC): Block Cloning merged by ksec on Wednesday July 05, 2023 @11:27PM (#36610747)
>having lost three major companies one after another from its list of users, sponsors, and developers a couple of years ago.

Which three?

Re:ZFS 2.2.0 (RC): Block Cloning merged by colonwqbang on Tuesday July 04, 2023 @05:08PM (#36592525)
Why don't you think it has legal weight? Or did you mean something else?

As far as know the point of EXPORT_SYMBOL_GPL was to push back on companies like Nvidia who wanted to exploit loopholes in the GPL. That seems to me like a reasonable objective.

Relevant Torvalds quote: https://yarchive.net/comp/linux/export_symbol_gpl.html

Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @05:15PM (#36592578)
Sure, and that alone isn't an unreasonable premise - as he says, intent matters.

But if you're marking interfaces as GPL-only, or implementing taint detection that means if you use a non-SYMBOL_GPL kernel symbol which calls a GPL-only function it treats the non-SYMBOL_GPL symbol as GPL-only and blocks your linking, it gets a bit out of hand.

Building the kernel with certain kernel options makes modules like OpenZFS or OpenAFS not link because of that taint propagation - because things like the lockdep checker turn uninfringing calls into infringing ones.

Or a little while ago, there was a change which broke building on PPC because a change made a non-SYMBOL_GPL call on POWER into a SYMBOL_GPL one indirectly, and when the original author was contacted, he sent a patch reverting the changed symbol, and GregKH refused to pull it into stable, suggesting distros could carry it if they wanted to. (Of course, he had happily merged a change into -stable earlier that just implemented more aggressive GPL tainting and thereby broke things like the aforementioned...)

Re:ZFS 2.2.0 (RC): Block Cloning merged by PlutoIsAPlanet on Tuesday July 04, 2023 @05:21PM (#36592646)
The Linux kernel has never supported out of tree modules like how ZFS works out of tree.

All ZFS needs to do is just have one of Oracles many lawyers say "CDDL is compatible with GPL". Yet, they Oracle don't.

Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @05:31PM (#36592746)
"All."

It's explicitly not compatible with GPL, though. It has clauses that are more restrictive than GPL, and IIRC some people who contributed to the OpenZFS project did so explicitly without allowing later CDDL license revisions, which removes Oracle's ability to say CDDL-2 or whatever is GPL-compatible.

So even if someone rolled up dumptrucks of cash and convinced Oracle that everything was great, they don't have all the control needed to do that.

Re:ZFS 2.2.0 (RC): Block Cloning merged by Dylan16807 on Tuesday July 04, 2023 @05:17PM (#36592599)
To have legal weight, it has to be a signal that you're implementing something that is derivative of kernel code. That's the directly stated intent of EXPORT_SYMBOL_GPL.

But "call an opaque function that saves SIMD state" is obviously not derivative of the kernel code in any way. The more exports that get badly marked this way, the more EXPORT_SYMBOL_GPL becomes indistinguishable from EXPORT_SYMBOL.

Re:ZFS 2.2.0 (RC): Block Cloning merged by colonwqbang on Tuesday July 04, 2023 @05:52PM (#36592979)
I see it as just a kind of "warranty void if seal broken". Don't do this or you may be in violation of the GPL. Maybe a legal court in $country would find in your favour (I'm not convinced it's as clear cut as you imply). Maybe they would find that you willfully infringed, despite the kernel devs clearly warning you not to do it.

The main "legal effect" I see is that you are not willing to take that risk, just like Oracle isn't.

Re:ZFS 2.2.0 (RC): Block Cloning merged by Dylan16807 on Wednesday July 05, 2023 @07:33AM (#36598855)
If the goal was to prove someone saw the GPL, then the entire license enforcement mechanism wouldn't be in the kernel.

If enough symbols get restricted without valid copyright-based reasons, modules that legitimately have non-GPL licenses will have to lie to the kernel to get loaded. And that's a stupid situation all around.

Re:ZFS 2.2.0 (RC): Block Cloning merged by bakul on Tuesday July 04, 2023 @05:20PM (#36592633)
I suspect the underlying issues for not unifying the two have more to do with the ZFS design than anything to do with Linux. It may be the codebase is far too large at this stage to make such a fundamental change.
Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @05:48PM (#36592933)
I don't think so. The memory management stuff is pretty well abstracted; on FBSD it just glues into UMA pretty transparently, it's just on Linux there's a lot of machinery for implementing our own little cache allocating because Linux's kernel cache allocator is very limited in what sizes it will give you, and sometimes ZFS wants 16M (not necessarily contiguous) regions because someone said they wanted 16M records.

The ZoL project lead said at one point there were a variety of reasons this wasn't initially done for the Linux integration [1], but that it was worth taking another look at since that was a decade ago now. Having looked at the Linux memory subsystems recently for various reasons, I would suspect the limiting factor is that almost all the Linux memory management functions that involve details beyond "give me X pages" are SYMBOL_GPL, so I suspect we couldn't access whatever functionality would be needed to do this.

I could be wrong, though, as I wasn't looking at the code for that specific purpose, so I might have missed functionality that would provide this.

[1] - https://github.com/openzfs/zfs/issues/10255#issuecomment-620...

Re:ZFS 2.2.0 (RC): Block Cloning merged by bakul on Tuesday July 04, 2023 @06:07PM (#36593134)
Behlendorf's comment in that thread seems to be talking about linux integration. My point was this is an older issue, going back to the Sun days. See for instance this thread in where McVoy complains about the same issue! https://www.tuhs.org/pipermail/tuhs/2021-February/023013.htm...
Re:ZFS 2.2.0 (RC): Block Cloning merged by rincebrain on Tuesday July 04, 2023 @06:23PM (#36593277)
That seems more like it's complaining about it not being the actual page cache, not it not being counted as "cache", which is a larger set in at least Linux than just the page cache itself.

But sure, it's certainly an older issue, and given that the ABD rework happened, I wouldn't put anything past being "feasible" if the benefits were great enough.

(Look at the O_DIRECT zvol rework stuff that's pending (I believe not merged) for how a more cut-through memory model could be done, though that has all the tradeoffs you might expect of skipping the abstractions ZFS uses to minimize the ability of applications to poke holes in the abstraction model and violate consistency, I believe...)

Re:ZFS 2.2.0 (RC): Block Cloning merged by 5e92cb50239222b on Wednesday July 05, 2023 @03:16AM (#36596742)
Sun has picked the license specifically to make the project incompatible with the Linux kernel. If you start out with hostility, what treatment do you expect in return?
Re:ZFS 2.2.0 (RC): Block Cloning merged by depr on Wednesday July 05, 2023 @04:50AM (#36597621)
Surely you have a source for that that isn't someone else's comment.
Re:ZFS 2.2.0 (RC): Block Cloning merged by the8472 on Tuesday July 04, 2023 @05:57PM (#36593040)
Could the linux integration use dax[0] to bypass the page cache and go straight to ARC?

[0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt

Re:ZFS 2.2.0 (RC): Block Cloning merged by gigatexal on Tuesday July 04, 2023 @01:36PM (#36589947)
This is amazing. A detail of Netflix that, I a plebe, wouldn’t know if not for this site.
Re:ZFS 2.2.0 (RC): Block Cloning merged by ComputerGuru on Tuesday July 04, 2023 @01:47PM (#36590126)
Actually, Drew's presentations about Netflix, FreeBSD, ZFS, saturating high-bandwidth network adapters, etc. are legendary and have been posted far and wide. But having him available to answer questions on HN just takes it to a whole 'nother level.
Re:ZFS 2.2.0 (RC): Block Cloning merged by drewg123 on Tuesday July 04, 2023 @02:50PM (#36591051)
You're making me blush.. But, to set the record straight: I actually know very little about ZFS, beyond basic user/admin knowledge (from having run it for ~15 years). I've never spoken about it, and other members of the team I work for at Netflix are far more knowledgeable about ZFS, and are the ones who have managed the conversion of our fleet to ZFS for non-content partitions.
Re:ZFS 2.2.0 (RC): Block Cloning merged by gigatexal on Tuesday July 04, 2023 @04:06PM (#36591892)
Have they ever blogged or spoke at conferences about it? I soak up all that content -- least I try to.
Re:ZFS 2.2.0 (RC): Block Cloning merged by ComputerGuru on Tuesday July 04, 2023 @04:35PM (#36592217)
I’ve devoured your FreeBSD networking presentations but I guess I must have confused a post about tracking down a ZFS bug in production written by someone else with all the other content you’ve produced.

Back to the topic at hand, it’s actually scary how few software expose control over whether or not sendfile is used, assuming support is only a matter of OS and kernel version but not taking into account filesystem limitations. I ran into a terrible Samba on FreeBSD bug (shares remotely disconnected and connections reset with moderate levels of concurrent ro access from even a single client) that I ultimately tracked down to sendfile being enabled in the (default?) config - so it wasn’t just the expected “performance requirements not being met” with sendfile on ZFS but even other reliability issues (almost certainly exposing a different underlying bug, tbh). Imagine if Samba didn’t have a tubeable to set/override sendfile support, though.

Re:ZFS 2.2.0 (RC): Block Cloning merged by lifty on Tuesday July 04, 2023 @01:12PM (#36589580)
what do you use for the content filesystem?
Re:ZFS 2.2.0 (RC): Block Cloning merged by drewg123 on Tuesday July 04, 2023 @01:19PM (#36589685)
FreeBSD's UFS
Re:ZFS 2.2.0 (RC): Block Cloning merged by throw0101c on Tuesday July 04, 2023 @01:48PM (#36590147)
Any use of boot environments for easy(er?) rollbacks of OS updates?
Re:ZFS 2.2.0 (RC): Block Cloning merged by drewg123 on Tuesday July 04, 2023 @02:45PM (#36590989)
Yes. That's the bootonce thing I was talking about. When we update the OS, we set the "bootonce" flag via bectl activate -t to ensure we fall back to the previous BE if the current BE is borked and not bootable. This is the same functionality we had by keeping a primary and secondary root partition in geom a and toggling the bootable partition via the bootonce flag in gpart.
Re:ZFS 2.2.0 (RC): Block Cloning merged by xmodem on Tuesday July 04, 2023 @03:06PM (#36591252)
If you can share, what type of non-content data do the nodes store? Is this just OS+application+logs?