Theoretically you can mount rclone to a local directory, but realistically it doesn't work because in the absence of serverside borg it is very heavy on network IO and slows to a crawl, even for sshfs over a short physical distance. To use it remotely, you really need to have borg installed on a server. I currently do as you describe.

I've used it with an rclone mount, it seems to work fine. See https://github.com/rclone/rclone/issues/3641.