You can use Kerberos.
If you want to store user directories on kerberized NFS, users must use kerberos authentication.
I don't understand the objection to Kerberos tickets -- it's a single-sign-on system. You can obtain a ticket with a public key if you want, see e.g. http://www.h5l.org/manual/HEAD/info/heimdal/Setting-up-PK_00... I don't know how easy it aould be to use SSH keys, but I once designed an authentication system for a scientific facility that was based on PGP keys and a web of trust reflecting administrative/social structures in the days before things like Eduroam and (viable?) NFS4, but not before AFS, which has been successfully used in the same way for a long time.
I don't know why anyone would think "Kerberized" wouldn't involve Kerberos, but it's nothing specific to user directories.
OrangeFS (nee PVFS) is a userland parallel filesystem (with a kernel driver in Linux) that can use two types of public key security if you want that sort of thing, but I gather that's not widely used.
> krb5p Use Kerberos for authentication, and encrypt all traffic between the client and server.
As long as we're talking about TLS and IPSec, though, I'd point to Wireguard as maybe something viable for kernel use.