> never let people SSH in with a password

this seems to be a recent cargo cult rooted in the fact that people tend to choose weak passwords and/or reuse them, but there is technically nothing wrong about it and the opensshd defaults (MaxStartups) make bruteforcing reasonable username/password combinations unfeaseable.

A private key is just a very long password that is stored on disk and if there is no password on the key-file, it is stored there in plaintext an can be used by every other application (e.g. browser)

Much of security improvements these days are not just enabling better security, but making them easy and default.

Sure it's possible to: have a strong password, use it only for a single machine, keep it in a good password safe, never reuse it anywhere, have the discipline to not make it short/easy to type/easy to remember, change it every time a server is compromised, etc....

Or you could get all that for free with public keys. Yes a private key is like a very long password, but the important part is that it never leaves the client side and it's safe to use on every computer you want access to without the hassle of remembering one password per server.

If you access many machines, how will you know when one is compromised? How will you prevent an attacker from logging in as you?

Public keys really are the better way to handle user auth, less of a minefield for regular users.

> If you access many machines, how will you know when one is compromised? How will you prevent an attacker from logging in as you?

Even better is to use SSH certificates. That way you don't have to deal with authorized (usually permanent) keys.

Once the SSH CA is installed in the host, the client can generate temporary asymmetric keys and sign them with the CA key before every connection.

There are a few ways to set up this scenario, here's one using Vault:

https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-cert...

Not sure that's any better in security terms. It has some ease of use and central management benefits but also some significant complexity (setup and maintenance of a CA).

My setups just used puppet to manage a authorized key directory on each machine (basically one line of code), assuming you have a working puppet setup of course.

I'd consider either approach significantly more secure than passwords which is a much worse approach.

> Public keys really are the better way to handle user auth, less of a minefield for regular users.

I'll go along with that, thou i consider a secret that resides in my brain vastly more secure than one residing on my disk.

I 100% agree. But with ssh (ideally) the passphrase never leaves the computer you are physically touching.

With passwords you are sending it to remote computers where it could be compromised. Thus the standard practice of forcing all users to change their passwords when a server is compromised.

I've come to realize that if I don't use a password at least once per month, I will forget it.

...so, this isn't great advice either. Password vaults are important..... at which point, you can just use a complex unique password.

Okay, so use the 56 bits that you can store in your head as a passphrase to control the 256 bits (for ed25519) on your disk.

It’s not only bad passwords you’re battling against but social engineering and poor administrative practices.

I’m comfortable with telling people to turn off password auth on their sshd’s and treat password auth as the exception rather than the rule.

Also, it’s generally a nicer user experience.

There is a very significant distinction - a private key does not (or should not) leave the user's computer. A private key is very long password that is magically¹ never revealed to anyone during normal use, only it's accessibility is proven.

If the user does not verify the remote host's fingerprint (vast majority doesn't), they could be sending the password to an attacker / honeypot or just wrong server / password. And if they reuse passwords, server can record passwords and someone could try them elsewhere.

If you're using passwords, it is very likely the password was copy-pasted to an inappropriate host (or passwords common for many hosts). It is also very likely the first connection to a server from a given computer was without verifying the remote host (server fingerprint), users are likely try and accept all kinds of things if they connection doesn't work, and I've even seen people disable host key verification so they wouldn't be bothered when a server is reinstalled / ip is reused.

Local accessibility of private keys is a significant issue, but if a program can read .ssh/..., it can usually also alias ssh=store-pasword-and-ssh, so using password doesn't help that much, vs. it's other issues.

If you're using passwords for yourself, it may be manageable, but I never give other people ssh login access using passwords.

¹ using asymmetric cryptography

> A private key is just a very long password that is stored on disk

Absolutely 100% not. You're missing the most important aspect of PKI based authentication, the asymmetry. With symmetric authentication technologies, like passwords, the only way a user can authenticate is to hand over to the server (which might be malicious) everything that the server needs to impersonate that user.

Practically, this means that when I root the Linux server in your environment, and start up 3snake, I'm going to start collecting the passwords of every person who logs in, and I can use those passwords to escalate privileges around the network. This is not even a little bit possible in an environment where SSH key authentication is enforced.

That source you shared seems to show that there could be a problem with tainted RDRAND, since it is xored in at the end of the function, and the linked email thread has comments by "maintainer of record (me) for the Linux RNG" who "quit the project about two years ago precisely because Linus decided to include a patch from Intel to allow their unauditable RdRand to bypass the entropy pool over my strenuous objections.

From a quick skim of current sources, much of that has recently been rolled back"

At the very least, that looks like there might be something in it. I don't know crypto well enough to say for sure, but it looks like Linus needs to re-read rand.h from what that article says about the state of the comments and previous rolled back changes from Intel. It does make one wonder if Linus' arrogance and rudeness could be used as cover for a genuine backdoor.

It is xor'd because all entropy sources are xor'd. That's just the design of /dev/random.

And because of this, Intel or whoever would need to have weakened each and every entropy source you use to make anything of it.

That's not true, because the RDRAND entropy is mixed in last. So once you're under the assumption that RDRAND is nefarious, the microcode only needs to detect the rdrand-to-xor pattern to make the entire entropy pool predictable (for example: by setting the non-rdrand input to the xor operation to zero it could disable all other entropy sources).

A microcode backdoor capable of reading the existing entropy pool state is going to be a hell of a lot more powerful than a RDRAND backdoor, to the point of making a RDRAND backdoor worthless.

> braindead advice in there too, like disabling RDRAND because there might be a backdoor

Two sources of entropy xor'd will always be as strong as the best source. Done properly it's perfectly fine mixing rng sources you don't trust.

It is also important that the entropy sources are independent. For example, if RDRAND were defined in terms of /dev/xrandom, say by just flipping its bits, then doing

(secret bits) xor (bits from /dev/xrandom) xor (bits from RDRAND)

is very weak encryption. But of course, there is no reason to believe RDRAND is dependent on any of the usual random sources.

> (secret bits) xor (bits from /dev/xrandom) xor (bits from RDRAND)

> is very weak encryption.

It's as strong as the strongest source. This easily proven mathematically.

The mathematical result I think you are referring to is: any uniformly distributed random variable xored with any other independent random variable yields a uniformly distributed random variable. On the other hand, if we take the strongest random source in the universe, say X, and xor X with itself we get:

X xor X = 0

The result is just a sequence of zeros, which is not "as strong as the strongest source". In practice, people may take lots of different "random" sources as input to their random generators to make them "more" random. But if you don't take care to check if they are independent, you may have a problem.

On the contrary, the above computes exactly the identity cipher: your "ciphertext" is your plaintext.

Mathematical results will lead you to surprising conclusions if apply them in the wrong context!

Why do you think this is a common problem? I've never heard of anyone leaving private keys on the server. Usually the ssh-keygen is run locally on their machine and the public key is deployed on the server.

Oh, I've seen it plenty times when users want to SSH from server A to server B, they copy their private key from their workstation to server A.

A natural consequence of server admins disabling ssh agent forwarding.

All these “security truisms” tend to ignore the real world. When you remove features, people will work around it in a less secure way.

Oh boy... I need to step out into the wilderness more :-D

A result of people not understanding how public/private keys work. They just want it to work, and copy everything over to get what they want done.

These same people probably also joke about the marketing folks down the hall leaving post-it notes with their password on their monitor.

Security is the first casualty in the battle for convenience.

Urk. Wow.

The bigger issue is "Don't copy a private key. Ever. Period."

How would you even do this? Don't the various private key containers make this quite difficult?

It is just plain text in linux. Nothing hard to copy about it

What if you need to git ssh clone? Can you use your private key locally while your are currently connected to the a server?

Genuine question, this is why I have a private key on the server in out office so I can clone our private repos.

Can you use your local private key then?

Yes, you can pretty easily use "SSH agent forwarding", which grand-parent comment calls "the devil", but really it's not that bad when used selectively, and a lot better than copying the private key to the server.

If you're cloning a specific application, the server could have its own key with read-only access to the repo. But if you're using that for deployment, you could (in AWS) grant the instance access to specific S3 paths and pass a specific artefact URL to be downloaded. (Or something similar for other environments)

But maybe you've got a different use case? I'd start with "why are you using git directly on a (production?) server?"

I'm gonna out myself because nobody is giving reasons

Why is it bad to git clone on a production server? Let's assume I'm not an idiot please, just misguided. This isn't reddit.

Let's also assume I'm using an ssh key that is only used for the purposes of pulling the repo, no write access, never reused (deploy keys as the git* services call them)

It's really hard to see these tips as anything but parroting someone else if nobody throws down a reason

I don't think you're outing yourself exactly; this is a scale-dependent question. People will give different answers because their experience is coloured by the size of the systems and organisations they have experience with.

At small scale it's fine. Plenty of useful and valuable stuff runs on a handful of servers that have consistent identities etc and people manage them interactively. Process lives in people's heads or (better) a wiki.

At large scale it's a completely bonkers thing to do. There are tons of nodes; it doesn't make sense to mutate just one unless something has gone badly wrong. Interactive login to a production system should be triggering an incident or at least linked to one. Really you shouldn't be mutating systems in any kind of "manual" way because that kind of change is supposed to be locked down to authorised deployment methods. The current state of the system has to be legible to other team members and other teams, and the easiest way to do that is by looking at what's gone through the deployment process, which is usually navigable via a dashboard.

In the middle, it's possible you could have a deployment system that relies on "git clone" with a key on the instance. That would be a little weird because git is not a great way to store or distribute build artefacts. Not crazy though - could make sense in some situations.

So you already protected most of it - no write access and scoped keys fix most of the issues.

Other potential problems:

- bad checkout location may mean unexpected content is available via .git paths on the web

- anyone with access to the server can copy the key and have external access to both the history of the project and all new commits - they can see the PRs with proposed security fixes before they get merged

- repository may contain domain names, credentials and other things which don't need to be deployed, but can be useful for the attacker doing recon

- potentially exposing information about customers if they got mentioned in the history

It's not terrible to use git directly. There are just ways you can deploy a little bit better if it's worth your time investment.

  > Why is it bad to git clone on a production server? Let's assume
  > I'm not an idiot please, just misguided. This isn't reddit.

This cargocult ritual is for those who manage farms of identical servers. If you have a single production server with no load balancing, it's fine.

But be careful, Git does not handle file permissions well by default. If you have a directory or file that needs special permissions (like a cron job script) then you should set `core.fileMode` to `false` in your git config.

> This cargocult ritual is for those who manage farms of identical servers.

Then it's not cargo-culting, since for farms of identical servers manually checking out a repo is going to cause you pain.

It _is_ cargo-culting when publications begin to recommend it as standard practice, without explaining the reasoning. Because at that point either the publisher is doing it without understanding why, or the reader will follow the ritual without understanding why.

If you are running a modern proper devops environment you shouldn't be git cloning anything on a production sever, nor would production servers be using the same ssh keys to do that via whatever production server login you configured to reach out externally as the keys from a local sys admin dev machine to ssh into to the production server, which hopefully if you're running anything significant in production is through a proper IAM account or service account through a command line tool authing you to a certain level of underlying vm/node access to all production services in that rbac role.

To be hotly debated but regardless of the most secure production deployment environment you should have production environments like layered images and all required installs on a private company image repo where the company vpc deploys through a build server that deploys image tags to a private company image repo in which an ingress pulls down from after running things like clair and what have you to make sure at the very least you have the latest library versions running with the most recent patches on stable and can deploy replicas as across many servers/nodes through an intermediary deployment architecture whether it be k8s or nomad (which also allows for VMs not just containers) in which you could standardize and highly specify seccomp profiles app armour and any custom kernel modules or what have you.

Anyone using the same server to git clone (messing around in a private development box) to deploy production services would be a no go for me.

Any gitclone should be automated from a jenkins or similar build where org repo keys are authed before pushing to a private company repo (which also required with)

The only thing more annoying than the comments scoffing at how basic the advice is the reflection of how basic the production deployment techniques of the scoffers are. Nothing should be so non standardized in production as got cloning on a prod server ad hoc.

Ssh keys on a server seem an irrelevant situation to me in any production scale server I've worked in.

You'd be surprised. I worked at a startup and our entire deployment system was based on "git pulls" onto each production node, recompiling binaries, etc. Yes, it was crazy and also slow.

If you are able to use https instead of ssh, I find "deploy keys" quite handy for this scenario. Gitlab/Github provide these as essentially a temporary https password for selective read-only access to a repo or group of repos with an expiry date. The main downside I find is incompatibility with anything that needs dependencies and expects git over ssh... but then I try to avoid creating the scenario where repos must be built or assembled on each server.

What I would really prefer is to be able to use git over ssh with U2F, i.e a hardware key, in place of a private SSH key this should work the same from the server as the client. U2F is already in openSSH but I am not sure how long it will take before it is commonly available and added to git hosting services... i'm also not sure if the protocol will work through the terminal to an openSSH client on a server.

I already use hardware keys for OTP with SSH (yubico pam) which makes doing ssh between servers secure and easy without private keys, and without client software compatibility issues since it's just a keyboard-interactive mode as far as SSH is concerned... In fact if you are also hosting your own git service you could use this for git cloning over ssh right now and not bother waiting for U2F.

> Also SSH agent forwarding is the devil. Don't do it.

Care to elaborate? First time I hear this.

Others already explained why. Matrix.orgs postmortem includes a real world exploitation: https://matrix.org/blog/2019/05/08/post-mortem-and-remediati...

If you forward SSH agent, anyone with privileges to your sockets can use the agent. So for example when you log into some production machine, a process waiting on that machine connects to your agent and has access to... everything. (If you're agent caches key password you won't even get notified)

Assuming your local agent silently signs all requests. This attack doesn't work if e.g. your Yubikey is configured to require a physical tap for every authentication.

Using ssh-add -c when adding keys means it asks for user confirmation (but can still cache the passphrase) each time key use is requested. This can mitigate some of the risk here in the absence of other solutions.

This is a good theory and I mostly agree but if you disallow password logins, private keys on servers and agent forwarding - how do you copy things server to server without going through your slow and possibly remote local connection?

Magic wormhole and friends are pretty good at doing this securely, especially between servers that can talk to each other without relays.

See also croc

python3 -m http.server —bind 0.0.0.0 8080 is my favorite command

If we're talking about security, then exposing your files to anything with network access to your server sounds like a terrible, terrible idea.

I 100% endorse this method of securely moving files way way way above using things like agent forwarding and throwing keys on servers.

It's shocking to me how people overlook HTTP as a tool for passing files around.

You say that having private keys on servers is bad, and you say that using SSH agent forwarding "is the devil."

How, exactly, do you expect people to do things like use git or pivot from Server A to Server B over SSH?

HTTP is a thing.

Yeah, when pushing up files from a client to a server, or pulling down files, SSH makes sense, but for server to server, there's no need to use something with a full control channel for a simple file transfer.

As to your question on how to use jump hosts, that's what -J is for.

ssh -J ServerA ServerB

How would you automate a job where a server uploads something, say a backup, to another server, without a private key?

Standalone private key for a separate account that has write-only access to the backup area and little else?

In this case the key would act more like an access token for a specific operation that you're trying to perform, rather than as a key to a generally usable account.

Yes that makes sense. I think grandparent could have been more precise. I assume he mean _personal_ private keys.

I think he's talking about

    $ ssh-keygen -t rsa
    $ mv .ssh/id_rsa.pub .ssh/authorized_keys
    $ cat .ssh/id_rsa
    $ exit
    logout

In theory, I know Genode and other capability based systems could be answer this question... but not in practice. I'm off on that tangent seeking answers. Thanks for the "hacking prompt"

Use HTTP, there's absolutely no reason to be using SSH if you're just transferring files around.

> Never let people SSH in with a password...

Why not add libcrack (constraints) and auto-expiration of passwords instead of locking people out completely?

Also adding fail2ban is a very effective solution to prevent brute forcing.

> and for the love of god, stop leaving private SSH keys on servers

Why not? Unencrypted private keys are dangerious, OK, but I fail to see the problem with a good password protected private key?

So maybe if you add constraints and force rotation you can eventually get a secure-enough system with passwords. You'll still be using a symmetric key rather than asymmetric, which means that compromises can be made worse (I get root on one server -> I get user passwords as they log in -> I have access to everything that they have access to now), and you've invested a lot of effort to secure it rather than just `PasswordAuthentication no`, `ChallengeResponseAuthentication no`, done.

Security is not a monolithic item with a single best practice. A horses for courses approach is better IMHO.

If you're exposing SSH to outer world, minimizing attack surface by removing users and/or passwords is good. OTOH, if you're not exposing these services outside, or putting them behind a VPN maybe with people connecting from unsecured locations, adding a password is always a good idea in my book.

A laptop maybe stolen, a workstation maybe breached if someone is not careful and losing a key create bigger consequence.

So maybe instead of a tug of war about which method is the silver bullet, we should discuss about which one is better, under which circumstances.

> Why not add libcrack (constraints)

If the user has a super secure password shared with a different, compromised service, libcrack will not detect that.

> and auto-expiration of passwords

Expiry results in passwords like: (prefix)Dec2020, (prefix)5, or cycling the last 2/3 entries. Keys can be relatively easily revoked and are guaranteed not guessable.

> If the user has a super secure password shared with a different, compromised service, libcrack will not detect that.

There's a module[0] for that (TM).

> Expiry results in passwords like: (prefix)Dec2020, (prefix)5

libcrack can enforce similarity and rotation checks too [1].

> or cycling the last 2/3 entries.

There's also another module[2] just for that.

> Keys can be relatively easily revoked and are guaranteed not guessable.

You're right but, a good password policy and infrastructure is not a simple straw man which can be set alight with a simple match. PAM can be a bit hard to understand but, once understood, it's pretty easy to create complex rules and flows.

If you get your workstation compromised somehow, you can always lose your keys. Then you need another password on top of your key to keep it encrypted.

At the end of the day, a good security policy is required. Keys, passwords, fingerprints and other identifiers are just tools. If you can design your defenses and moats well, you can secure your system with any method you want.

[0]: https://github.com/skx/pam_pwnd

[1]: http://www.linux-pam.org/Linux-PAM-html/sag-pam_cracklib.htm...

[2]: https://linux.die.net/man/8/pam_pwhistory

> libcrack can enforce similarity and rotation checks too [1].

How can it do that without the server storing plain text passwords?

Unless you're root, you enter your current password first for verification already?

Ah, ok, that makes a lot more sense. I was worried that the password history was stored somewhere.

> Unless you're root

I'm not sure I get this part, why does being root change things?

Normally PAM doesn’t ask for your current password during password changes if you’re root.

Also, root can change any user’s password without entering the current one.

No SSH keys, passwords, etc. Use Kerberos baby, SSO in!

SSH and Kerberos has been around for decades, but few companies have it implemented.

It completely removes the need for SSH keys and their related management, user/group management, etc., which is a huge security and administration win.

How do you do key exchange if no pw login in ssh (here helpdesk prints random pws)

You can bootstrap key exchange from a short-lived password. Not perfect but at least limits the attack time window.

Companies serious about security just have a trusted person hand out hardware with signed keys on it.

So is there a backup of the key on the hardware somewhere (in case it gets lost or destroyed)

Once you have exchanged the keys nothing stops you from creating new ones. You can for example register an additional backup Yubikey.

Small scale: You generate a keypair and give the public key to whoever is setting up your account on the server.

Large scale: You generate a keypair and give the public key to Vault or whatever, which signs it with the CA that all servers know to trust.

> RDRAND

Did you read your own link?