I use hashi corp's stack: Boundary + Vault for ssh credential injection

https://developer.hashicorp.com/boundary/tutorials/credentia...

it completely replaces the need for bastion hosts and ssh-agent at scale (I'm a hashicorp's paying customer)

I wrote a macOS app that supports using Vault Transit Engine keys with OpenSSH, as well as GnuPG and PKCS#11. (OpenSSH is supported via both the agent protocol and PKCS#11.) You can even use an Apple T2 (Secure Enclave) key for peer mTLS authentication to the Vault server. https://www.keymux.com/

Using this teams can share an SSH key without exposing the key; nor do you need to configure certificate PKI, use jump hosts, or otherwise change your existing software or workflows.

You can set up an ssh tunnel on localhost to any port on bar or github via foo.

As others have pointed out already, JumpHost or -J is the preferred way.

I agree!

The Github example complicates things, but the correct solution is actually another key:

Generate a separate private key on foo and place that one in github, ideally per project as a deploy key.

Please stop calling this the "correct" solution. It might work for you; it's not a general rule or widely accepted best practice.

Generating a key per host is just a different security model than key forwarding, and arguably a worse one if key forwarding is done defensively (i.e. forward a separate key, with only those permissions you would give to a key present on the connected host itself).

> Generate a separate private key on foo ...

That is a TERRIBLE idea, if foo can be compromised. Even if you secure the private key with a passphrase, it still needs to be loaded into foo's memory, by a binary resident on foo, which can be used to exfiltrate that private key.

If you're confident foo cannot be compromised, then the whole point is moot anyway.

Teleport is far and away the leading solution in this area — certificate based end-to-end bastion style topology with many nice features including support for kubectl.

Make an ssh connection to foo with a port forward. Then make an ssh connection to bar through foo. Keys stay on your machine. I'm pretty sure there's built in features to do this.

That won't work for a connection to github that's being initiated from foo.

The best solution in that situation is a separate key that's private to foo.

No, the best solution would be partial key forwarding of one or several separate keys only scoped to git access.

Better options:

c) use proxyjump (-J) to access bar (https://www.man7.org/linux/man-pages/man1/ssh.1.html), and/or

d) generate a new private key on foo and use that to access github.

You might think that's a crazy waste of time compared to forwarding, but it's actually much, much safer and actually only takes a moment:

    ssh-keygen -t ed25519
    cat ~/.ssh/id_ed25519.pub

and then just paste that in Github.

See also https://userify.com/docs/jumpbox for more jumpbox docs.

> d) generate a new private key on foo and use that to access github.

> [...] only takes a moment [...]

Not every org's policy allows adding unaudited ad-hoc SSH keys.

> [...] much, much safer [...]

Definitely not always, if the hosts you store these keys on are not as hardened as you local machine (or a hardware key connected to it).

> Not every org's policy allows adding unaudited ad-hoc SSH keys.

Then audit them and get them in the process. Agent forwarding is too big of a risk.

> Definitely not always, if the hosts you store these keys on are not as hardened as you local machine (or a hardware key connected to it).

Once you use agent forwarding, the keys are no longer protected on your local machine. (Ironically, this RCE is precisely because of the requirement to whitelist hardware keys!)

I need to read the source, I’m confused how -J actually works. Is the bastion doing auth and the downstream machines trusting? Or does it auth first and then forwarding a :22 connection from downstream back to localhost? Or something else entirely?

Just my understanding of the manpage and TCP forwarding (-L): an SSH connection will be established to the jump host, which then establishes a connection on port 22 to the destination. The local machine now has a forwarded connection to the destination and uses that to establish a second SSH connection between them.

Between local and jump host, there will be two layers of encryption. The jump host decrypts the outer layer, and the two ends the inner layer.

Just use a different, secure SSH Agent. I use gpg-agent and make it confirm every use of my private keys with me. Gpg-agent supports signing with a preloaded private key and nothing else, so it is immune to this attack and many others.