I generally use only a single node version, and therefore don't use the nvm command much, so I've solved the slow shell init by not loading nvm at all, and instead, just set the PATH.

    export PATH=~/.nvm/versions/node/v11.6.0/bin:$PATH

Downside is that you have to bump the version occasionally but it's worth the fast shell startup.