You can rsync the whole directory of passwords elsewhere and then connect there from your phone using SSH. If you're handy with `pass` you probably have an SSH client on your phone anyway (I use Prompt on iOS). Some people might think you're weird for using SSH from your phone though, fair warning.