I do ssh directly to the VM. It sits behind a VPN connection at aws. I need to make ssh connections from there to github. The key resides on a smartcard in my laptops usb slot. And that's where ssh-agent/ForwardAgent comes into play. I forward my local key to the remote VM.