This is one of the reasons my preferred implementation of Vi is Evil -- the TRAMP network layer in Emacs does exactly what you want: let you treat any remote file system as local. It even does multi-hop access through e.g. bastion machines. It is great.
I've never used any of those plugins, but the built in netrw has a directory browser that works.
On vanilla vim if you :e scp://host// you should get the built in netrw browser working over scp.
But I'm not really familiar with netrw other than just:
:e scp://host/path/file