SSH certificates avoid this issue and are used in real life. We make them with hashicorp vault, it works well and is really convenient.

Have to admit I'm not very familiar with SSH certs.

I'm generally just spinning up a fresh Ubuntu instance on EC2, so I have an entirely 'vanilla' SSH set-up. Small scale, so I don't especially mind the ritual of verifying the fingerprint. It's annoying though that it takes a few minutes for the EC2 system-log to be populated with the 'true value' of the fingerprint.

Huh, I had no idea there was even such a thing as SSH certificates!

Presuming you mean X.509 certificates, is this part of the standard spec (e.g. should work with OpenSSH etc)? Do you know if it works with PuTTY?

Here is a guide I've found useful in the subject, https://engineering.fb.com/security/scalable-and-secure-acce....

They are not based on X.509. I recommend taking a look at the CERTIFICATES section of the ssh-keygen man page, because I am unable to find a good reference on the web right now.

Big companies that care about security use short-lived certificates for SSH, not simple private/public keys.

> I've made a habit of manually checking the public-key when I SSH into a new EC2 instance using PuTTY. It is not obvious how to do this. Existing tooling just isn't set up with it in mind.

This is a not necessarily useful if you use key auth. The attacker can’t perform an useful MITM attack on the connection no matter what key you accept from the server. At best the attacker could let you log into their server, but presumably you’d notice that you’re logged into the wrong server.

I guess there are a few corner cases where this might help, such as blind scp uploads.

Surely the attacker can serve up an SSH server implementation that forwards the connection to the intended server while recording and/or modifying the decrypted traffic as desired? To the user everything would appear as normal (expect perhaps for some additional latency) but their data would be exposed to the attacker.

If there were no way to perform a MITM attack because you would notice that you were connected to the wrong server, the whole system of keys and fingerprints would be pointless. So would the CA system for HTTPS.

> Surely the attacker can serve up an SSH server implementation that forwards the connection to the intended server while recording and/or modifying the decrypted traffic as desired? To the user everything would appear as normal (expect perhaps for some additional latency) but their data would be exposed to the attacker

The attacker can’t do this because the attacker does not have your private key. If the attacker proxies your authentication to the remote server, they’ll only be back to square one as the connection is now encrypted with keys the attacker does not hold.

This is only possible if you use password auth.

> If there were no way to perform a MITM attack because you would notice that you were connected to the wrong server, the whole system of keys and fingerprints would be pointless. So would the CA system for HTTPS.

It’s much easier for an attacker to create a malicious copy of a website they have access to than a server they don’t.

The phishing opportunities offered by a SSH mitm attack like this are rather limited.

> The attacker can’t perform an useful MITM attack on the connection no matter what key you accept from the server ... The attacker can’t do this because the attacker does not have your private key.

Why does the attacker need my private key? If he has his own private key and I accept it then he can successfully perform a MITM attack and proxy the connection to the real server.

If not, then what is the purpose of key fingerprints and the web CA system?

There is no need to involve "malicious copies" of servers or sites - you just have to intercept the data in transit between the user and the real server. If the client connects to the attacker (accepting his key) and the attacker connect to the real server, both client and server believe they are talking to each other but they are actually both talking to the attacker, who passes the data on. For HTTPS there is a handy tool[1] available to do this. If there isn't already one for SSH, the same principle still applies.

[1] https://mitmproxy.org/

> Why does the attacker need my private key?

To authenticate as the user on the target server.

> If he has his own private key and I accept it then he can successfully perform a MITM attack and proxy the connection to the real server.

That's not the case.

If you accept the public key offered by the attacker's machine-in-the-middle, then half the attacker's work is done, as they've tricked you into connecting to their machine. The other half of their work still remains though: they need to connect to the target server, impersonating the user.

They can't do this, as they don't have your private key. It doesn't do them any good to try to generate their own keypair, as it won't be recognised by the server. (Roughly equivalent to guessing at a password.)

I'm not sure if I've been very clear. There are plenty of explanations of public key crypto out there better than I can provide here.

It's a major advantage of using public keys to authenticate users, rather than passwords. (Servers are always authenticated using a public key.) If we were using passwords, then in the situation we've described above, it would be possible for the attacker to connect to the target server, impersonating us, and enabling them to set up a man-in-the-middle. The attacker just needs to capture our password when we send it to them. Of course, as they now have our password, they might do plenty else besides.

> what is the purpose of key fingerprints and the web CA system?

It's to ensure the target machine really is the expected target machine.

As you say, the web uses certificate authorities. SSH supports a similar approach, but the 'conventional' SSH solution is to manually check the server's fingerprint (which has the effect of checking the server's public key).

> For HTTPS there is a handy tool[1] available to do this. If there isn't already one for SSH, the same principle still applies.

That's just a web proxy, there are many of these available. If such a proxy is used maliciously to attempt to intercept an HTTPS connection, the connection will fail the browser's certificate-authority check, and will be terminated. (At the very least, the user will be shown a scary warning popup, but sensible modern browsers may just refuse the connection entirely, as users tend to unthinkingly click through such popups.)

These proxies can only work with HTTPS if the (self-signed) certificate used by the proxy, is added to the browser's list of trusted certificates. [0]

[0] https://docs.mitmproxy.org/stable/overview-getting-started/

Ah, good point, thanks. I misread the first sentence of the comment I was responding to as being about the server's public key, so I was assuming password authentication.

With pubkey authentication (or rarely-seen-in-practice authentication with client SSL certs on the web) the attacker couldn't impersonate the client.

Right, but it's still pretty disastrous if the attacker succeeds in tricking you into connecting to their machine, so you still need to verify the server's fingerprint.

> This is only possible if you use password auth.

It's not possible with password auth, either.

If you check the server's SSH fingerprint on first connection, you're safe from man-in-the-middle attacks, regardless of how you authenticate the user. If you fail to check the server's fingerprint (the way most people use SSH), then you haven't verified the identity of the server, and again this is regardless of how you authenticate the user.

There are other advantages to using a public key to authenticate the user, though. [0]

[0] https://security.stackexchange.com/a/69408

> It's not possible with password auth, either

It is, if you screw up with TOFU.

> If you fail to check the server's fingerprint (the way most people use SSH), then you haven't verified the identity of the server, and again this is regardless of how you authenticate the user.

This has different ramifications depending on how you authenticate.

> It is, if you screw up with TOFU.

If you simply forget to check the fingerprint, you've got a problem, yes, and this is a risk if it's expected to be done manually. Most users simply can't be bothered, despite that their SSH security depends on it.

As I said though, if you screw up the TOFU check, it's game-over either way: you've failed to verify the identity of the server. If you perform the TOFU check properly, then you have verified the identity of the server. Again, this is regardless of whether you use a password or a public key to authenticate the user.

> This has different ramifications depending on how you authenticate.

Right. With public key authentication, the server is never sent the user's secret (their private key), unlike with password authentication, where the server is entrusted with the user's secret (their password).

This could of course be significant if the attacker is able to capture the user's password, perhaps for the purposes of setting up a man-in-the-middle attack, but I think it's reasonable to treat it as game-over if an attacker has tricked a user into connecting to a machine under the attacker's control. HTTPS works this way; we should apply the same thinking to SSH.

> The attacker can’t perform an useful MiTM attack on the connection no matter what key you accept from the server.

True, thanks to public key crypto.

> At best the attacker could let you log into their server

This is another way of saying the machine you end up trusting could be under the control of an attacker. So yes, that's a serious concern.

> presumably you’d notice that you’re logged into the wrong server

Not if it's a fresh Ubuntu instance. Even if not, I'd expect SSH security to answer that definitively, on principle. SSH shouldn't be any sloppier than HTTPS.

Maybe I'm being naive here, but if you spin up a new instance and immediately SSH into that via the given public IP, isn't the likelihood that someone could have meaningfully injected themselves between you and that previously unknown IP address vanishingly small?

Just monitor and mitm all ssh connections. It’s really easy.

Right. The attacker might have compromised my local network, for instance.

What's absolutely incredibly frustrating is that it's absurdly common to ignore SSH fingerprints even if they're staring you in the face. I just spun up a Fedora 32 cloud image on RamNode over the weekend and the default image oh-so-helpfully displays the SSH fingerprints as part of the MOTD so you can verify it just looking at the login prompt on console. Except apparently the script that generates that runs before the SSH keys are regenerated on the first boot of the image, so the fingerprints displayed don't match the keys on disk. Either no one else has noticed this bug yet or nobody has been bothered enough to fix it.

There is a solution though, SSHFP records for the host. I just set them up after catching that issue just because of how frustrating blindly trusting on first use is. Now I can SSH to my servers even without anything for them being in known_hosts and it checks for SSHFP records and verifies the fingerprint and DNSSEC protects the SSHFP records.

SSHFP requires you to DNSSEC-sign your zones, which virtually nobody in the technology industry does, for a variety of reasons, some involving security.

A more straightforward solution that doesn't require you to hitch your wagon to a moribund infrastructure overhaul is to use SSH certificates. SSH certificate infrastructures have other benefits beyond resolving the introduction problem; for instance, they're easier to manage than keys once set up, and, perhaps most importantly, they make it possible to issue short-lifespan keys based on single signon, which is much safer than having everyone register a canonical personal key.

I would recommend using the original tooling.

The ssh CLI shows the fingerprint, and explicitly asks:

  $ ssh example.com
  The authenticity of host 'example.com (1.2.3.4)' can't be established.
  RSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
  Are you sure you want to continue connecting (yes/no/[fingerprint])?

That's roughly what I've ended up doing. [0]

Both OpenSSH and PuTTY ask you to confirm the fingerprint, but PuTTY's fingerprint scheme is MD5/hex, whereas modern versions of the OpenSSH client use SHA256/Base64.

[0] https://serverfault.com/questions/996828/#comment1296514_996...

Here's an approach that I'm using with small VPS providers. I'm using web remote access to install Linux (from pristine distribution ISO). That web remote access uses HTTPS, so I can be sure that all input/output is E2E protected from provider to my computer. After I install Linux, I boot it first time and access it using that HTTPS remove console. Then I'm running something like ssh-keygen -l -f /etc/ssh/ssh_host_ed2519.pub -E md5 (I might be wrong about this command, check mans before use, also it does not require root, so login under ordinary user) and I can see md5 hash of server ssh public key. I write it down (or rather print-screen), connect with Putty and verify it. I believe that this method provides as much chain of trust as possible.

That's the approach I use with Linode. Unlike EC2 there's no browser-viewable system-log that shows the public key, but it can be retrieved using a web-based terminal session.

To find the fingerprint in PuTTY's obsolete format, the command I use is equivalent to the one you've given. It was provided in the ServerFault answer I linked:

     ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_ed25519_key.pub

I don't like comparing fingerprints by eye, but PuTTY doesn't make it easy to do otherwise. This issue doesn't arise with PowerShell's OpenSSH client, where I can just copy the fingerprint (in the new format) to the clipboard. (It lacks features like mouse support though, so I prefer to use PuTTY.)

To find the fingerprint in the new format for OpenSSH, I believe the command is:

    ssh-keygen -l -f /etc/ssh/ssh_host_ed25519_key.pub

> I believe that this method provides as much chain of trust as possible.

Yes, can't do better than that. If your HTTPS session with your VPS provider has been compromised, it's all over anyway.

Thank you for asking serverfault question and writing about it here.

When I started to work with EC2 and SSH few years ago I encountered this problem. However I was overwhelmed with the stuff I had to do for actual business and had no energy to investigate.

Next question is, how do we protect against this attack without waiting for few minutes and making all those clicks to find the fingerprints every time we launch an instance?

Someone mentioned SSH certificates. My understanding is that it's the 'proper' solution, but is more work to set-up. Whether it's worth it will presumably be a question of scale.

The instance itself needs to get its SSH host key signed, or upload its fingerprints somewhere, on first boot. This is easy enough to write, but making it any more secure than plain old TOFU is tricky.

> making it any more secure than plain old TOFU is tricky.

To refer again to my complaint about this term: this isn't meaningful. It's not clear whether you mean trust blindly on first use (total failure to cryptographically verify the identity of the server) or manually check SSH fingerprint on first use (secure but inconvenient).

Both get called TOFU. One meaning of the term is a valid means of establishing a secure channel, and the other is cybersecurity negligence.

I'm not just being obtuse here, I really don't know which you mean. Manually checking fingerprints on first use seems like a pretty solid approach to SSH security, if you can trust the client to actually perform the check. Short-lived keys are the obvious improvement that this approach doesn't really support. Is that what you meant?

Where do you get the fingerprints to check against? Three intuitive but wrong answers:

a) By spawning the instance with a script that will post them to some API. Now you are just blindly trusting whatever gets posted to that API. If your provider offers instance identity documents, you could sign the request with that, but an attacker who can MITM within your VPC can presumably impersonate anyone to the instance metadata service. And if MITM within the VPC is not in your threat model, then blind trust (from another host in the VPC) is fine.

b) By having an SSH CA service connect to the instance to grab its public key. The CA is extending the same blind trust as the admin would have in doing it manually.

c) By having generated the host keypair somewhere other than the host. Now you have key material in the wrong places.

I don't think there's such a clear-cut line here. "Verify" just pushes the blind trust somewhere else.

I can think of some potentially better answers, for example write the public key to a block device, detach it from the new host, attach it to a verification service host and read it there. But this is kind of just reinventing networking... why do we trust the cloud provider's SAN more than its SDN?

> Where do you get the fingerprints to check against?

Any serious cloud/VPS provider will offer a way, leveraging HTTPS.

On EC2, the Linux distro writes the SSH fingerprint to the instance's 'system log' which can be securely viewed over the AWS web dashboard.

On Linode, you can use a web-based terminal session (over HTTPS of course) to run a command to show the fingerprint.

I've discussed this in other replies in this thread.

> "Verify" just pushes the blind trust somewhere else.

Not really. If my cloud provider is compromised, it's game-over anyway. Provided there's a secure channel to communicate the fingerprint, and provided we really do check the fingerprint (most people just don't bother), we should be ok.

> write the public key to a block device, detach it from the new host, attach it to a verification service host and read it there

Yep, that's pretty much the Amazon approach!

Caring about MITM inside the VPC already assumes cloud provider compromise.

It's not inside the VPC, I'm connecting from the outside. I agree there's little need to check fingerprints for VPC-internal sessions.

Note that you've introduced the additional measure of "inside a providers VPC". Which indeed is a solution, but not something every environment provides and not everyone uses.

How many organizations are paranoid enough to care about MITM but not enough to have private networking?