Interesting that this was tested at Framestore (the visual effects company), they used to be a Lustre shop years ago then moved to commercial NFS servers and must be back onto commodity hardware now.

I could see a use for both - using Lustre as a shared SAN for their flame/smoke/etc suites and using pNFS for workstation/render-node shared storage. at least that's how i've deployed things in the past.

i could see this replacing isilon (which is built on freebsd) for the later use-case :)

Very alpha code, YMMV, buyer beware. I don't want anyone to try this and be turned off FreeBSD if it doesn't work well.

The author only has 100 Mbit networking, so performance bottlenecks may not be well-explored.

This seems pretty cool in principle, but I got as far as a single nonreplicated MDS before I lost interest in trying it out at this stage.

Admittedly, my background is in HPC where metadata performance and reliability requirements would preclude using something like this. Does anyone who is a more familiar with FreeBSD than I am (which is pretty much anyone who’s used it in the last 10 years) have any more information on long their term plans for at least replicated block storage failover or cache coherency in an active/active MDS setup?

pNFS in FreeBSD is one developer's toy project. I don't know of any widespread interest or plans for enterprise features.

It will be fairly amusing to see this in a Billions dollar product line like NetApp or Dell EMC Isilon.

I can't find any photos of Rick Macklem's setup but it was developed in a fairly.. steampunk fashion.. on several obsolete i386 laptops with <4GB RAM.

Panasas, who open sources pNFS, is a serious scale company competitive with Isilon and NetApp

https://www.pcmag.com/article2/0,2817,2135264,00.asp

https://www.panasas.com/

Since Panasas contributes a fair amount to FreeBSD, perhaps they are happy as well.

eg https://people.freebsd.org/~rpokala/2017-03-24.nvdimm.txt

Ravi never actually got around to committing that NVDIMM work :-(.

I believe NetApp has had shipping pNFS support in ONTAP for quite some time now. I don’t know how many active users though.

Why the hate? Just because FreeBSD supports the idea of making a commercial product from the code isn't a reason bag on Rick.

I didn’t detect any hint of hate in the OP’s comment, more appreciation and irony.

Isilon hasn't used the FreeBSD nfsd since 2014 (OneFS 7.2). (And prior to that, it was a heavily modified version.)

What is pNFS?

Parallel NFS, which was introduced in the NFS v4.1 protocol.

Thanks!

I’m still surprised that NFS doesn’t have a TLS option. Or a way to grant access without relying on IP whitelists.

> Or a way to grant access without relying on IP whitelists.

You can use Kerberos.

But then you must use Kerberos authentication (password only) and can't use SSH public key authentication.

Thats simply not true. GSSAPI allows you to forward kerberos tokens over ssh, with or without passwords or keys.

You misunderstood. The problem is not forwarding Kerberos tokens, the problem is obtaining Kerberos tokens. Unless some extensions are employed, you need to use password authentication. Even if kerberos extensions are employed, you must still use kerberos auth instead of ssh public keys. If you try to use ssh public key authentication (instead of telling ssh to use kerberos authentication) you can't log-in to the machine because you can't access your home filesystem, which is kerberised. In fact, depending on your configuration, you might not be able to log-in at all since the user's filesystem might not be mounted at all (waiting for the automounter).

If you want to store user directories on kerberized NFS, users must use kerberos authentication.

Kerberos actually operates on keys. The nfs service key is typically a random one stashed in the keytab.

I don't understand the objection to Kerberos tickets -- it's a single-sign-on system. You can obtain a ticket with a public key if you want, see e.g. http://www.h5l.org/manual/HEAD/info/heimdal/Setting-up-PK_00... I don't know how easy it aould be to use SSH keys, but I once designed an authentication system for a scientific facility that was based on PGP keys and a web of trust reflecting administrative/social structures in the days before things like Eduroam and (viable?) NFS4, but not before AFS, which has been successfully used in the same way for a long time.

I don't know why anyone would think "Kerberized" wouldn't involve Kerberos, but it's nothing specific to user directories.

OrangeFS (nee PVFS) is a userland parallel filesystem (with a kernel driver in Linux) that can use two types of public key security if you want that sort of thing, but I gather that's not widely used.

You can use kerberos for authentication, and also encrypt traffic [0].

> krb5p Use Kerberos for authentication, and encrypt all traffic between the client and server.

[0] https://wiki.debian.org/NFS/Kerberos

NFS tends to be living in the kernel, which sounds like a terrible place for TLS. That said, IPSec lives in the kernel to some extent, you could layer NFS on top of ipsec....

Linux and FreeBSD have both got kernel TLS support (not sure if freebsd got merged yet but Netflix have used it fir a while)

Netflix's TLS support is an extremely limited hack that suits their performance needs. It does not support initial session negotiation or rekeying. They do the former in userspace before handing off a symmetric key to the kernel, and drop connections in the latter case, relying on the client to reconnect. There's no chance it will be merged to FreeBSD; it's not a general solution.

As long as we're talking about TLS and IPSec, though, I'd point to Wireguard as maybe something viable for kernel use.

The Linux kernel TLS support does the same.

For what it's worth, NFS-Ganesha is a serious userland server.

NFS had (presumably still does) a private(?) key security option, but the Solaris fileservers I last used had given up on that in favour of Kerberos.

Red Hat worked pretty closely with NetApp and their implementation of pNFS. I've used it and it is very nice for mitigating any downtime while you perform maintenance on a controller.

How does a parallel filesystem make maintenance easier? Unless it's specifically redundant, I'd expect the opposite.

Is NFS viable at all with untrusted clients on a LAN? Seems it requires all machines on a LAN to be trusted, with proper user IDs.

It's fast though.

Apart from Kerberos, you can use it in a safe way in all-squash mode, where access from certain IPs are always interpreted as a certain UID. So if you're e.g. distributing built code to your network, doing all-squash to a user with read-only permissions would work.

This requires a trustworthy routing config (or a way to write files using something other than NFS) but allows untrusted clients as long as someone is keeping them out of trusted subnets.

Note that unlike all-squash, root-squash is not a meaningful security measure, since an untrusted root user can just become whatever UID they want. It's simply a (useful) protection against mistakes as root.

I've actually implemented NFS servers professionally (for my sins) and used various competitor implementations. It's possible with the mechanisms others have mentioned, but ultimately each vendor has its own specific ways to circumvent the "Kerberos is annoying and often not well configured at the client" issue[0]/allow various 'hacks' to be deployed, so if you're interested in a specific vendor's product (e.g. ONTAP) the best option is to look through their online manual for the specific CLI/Web GUI features, usually around root squashing and the like.

[0] I actually really like it, but the reality and customer feedback is what it is...

In what way are implementations not inter-operable? The SunOS and Linux implementations, at least, coexist happily (modulo ACLs). Do others fail to implement the standard?

Is Kerberos even an option in an environment with Mac OS & mobile phone clients, as well as Windows?

It is, but (IMHO):

1. There aren't many NAS devices that gracefully interact with both Active Directory and Kerberos (if you care about that, for both SMB and NFS access to the same storage). The best that I know that is still commercially available is ONTAP (I was not employed by them, but a competitor). I don't know much about its internals, but was impressed with the features, team, my own play with it, etc. It's also the most expensive. If interoperability with AD is not an issue, then disregard.

2. Mac OS X support for NFS v4 and its variants is abysmal. To the point that I could craft packets that would cause a reboot on the latest OS (which to me yells probable 0-day... but hopefully fixed since then and security analysis is not my expertise). Their SMB client is really good though, second to Windows in terms of keeping up with features, using them correctly, etc.

3. What's the use case for the mobile devices? The NFS/SMB clients I've seen for mobile are clunky as hell, but it was also never my focus.

While still only my opinion, this is true up to early 2017. I've since moved to another industry and don't actively research it anymore.

I've used krb5 and nis in an osx environment with NFS as file store. My basic opinion is that osx is not about unix legacy and they move as desired. Running on osx is agsinst a moving target.

Unsuitable for production.

On my home Mac, /Users lives on kerberized NFS. It was painful to configure but it does work.

I have used kerberos on Windows too without issue, though not NFS so not sure how good its NFS client is. (It does have one.)

Mobile phones using NFS with kerberos? I doubt it.

You can set up Kerberos authentication, but I've never actually seen this done in the wild.

Most setups trust based on subnet.

It's a lot more common than pNFS, which I've never seen. You really don't want anything else unless you have tightly-coupled, well-managed clients.

If you can trust the existing machines, but are worried about new machines sniffing and spoofing, you can set up IPsec between the NFS clients and the NFS server.

With kerberos authentication you can also require traffic encryption.

> krb5p Use Kerberos for authentication, and encrypt all traffic between the client and server. [0] https://wiki.debian.org/NFS/Kerberos

I wish there was a middle ground where you could issue a ticket to the server and not have to issue one per user but still have the traffic encrypted between NFS client and NFS mount.

Hm, would NFS over WireGuard do what you want? Or over stunnel, maybe?

Since WireGuard securely attaches an IP address to a public key, NFS over WireGuard may actually be a viable solution.

Thanks for the suggestion I'll look into it!

Perhaps you'd like to read up on NFS basics.

The short answer is, you can use it with either loose or tight authentication. The "LAN" bit of your question is largely irrelevant.

Subversion? oh man, what a blast from the past.