It's pretty standard practice to do this post-exploitation.
Further this requires you to regularly check your log files for suspicious activity, which is way more work than just rotating your ssh keys - which can be easily automated. Running a script that will automatically rotate your ssh keys on all servers in your .ssh/known_hosts is trivial.
Also, rotating your ssh keys is something that has a chance to prevent intrusion, whereas if you see something in your logs it's already too late.
It's not that simple. In a larger/mature environment you'll have log aggregation where the initial login is close to certain to be forwarded before it can be messed with. (Unless someone can log in, escalate, kill the right daemon, and somehow prevent monitoring from noticing a missing endpoint - all before the log gets forwarded)
And that's just host logs, without the networking, potential forwarding, etc.
As for endpoint logs, yes, a privileged attacker could disrupt them. But honestly, even with regards to your endpoint, attackers often don't disable logging - though I do see it, for sure. For "blessed" logging like Windows Event Log you'll have an even harder time - they do take measures to protect the files on disk, even against privileged attackers, and supported methods of deleting the event log actually themselves generate a "Someone deleted the event log" event, which I would highly recommend you watch out for :)
The simplest advice for dealing with this is to ship logs off of the device ASAP and to make sure that disrupting the service requires privileges.
Trying to prevent a determined privileged attacker from doing something is an exercise in futility, since it's impossible in every sense of the word. If they have total control over a system, they can do whatever they want, even if you put up a bunch of stopgaps.
Most post-exploitation frameworks (prominent example: DanderSpritz) have modules to remove stuff from the windows event logs without leaving traces.
It's pretty basic stuff.
That's not really true at all.
> since it's impossible in every sense of the word
Ok? Lots of things are impossible, and lots of those things are also still very very hard and costly. Hash collisions aren't impossible, and yet here we are, with a world hinging on them being very hard.
> , they can do whatever they want
Not really.
> Most post-exploitation frameworks (prominent example: DanderSpritz) have modules to remove stuff from the windows event logs without leaving traces.
You realize that:
a) DanderSpritz's logic to bypass the event log was a huge deal
b) It's literally an NSA leaked exploit????? Like are you kidding me using an NSA developed exploit as "pretty basic stuff" lol
Sorry but I think I'm gonna stop responding here.
It must have been very expensive and innovative, does that make it hard to copy into your scriptz folder?
Sorry, but this entire thread is nonsense, and it's just a clear demonstration of a lack of threat modeling and frankly a lack of understanding of what attacker capabilities are.
You can try to play games with ensuring logs leave the system, but everyone takes shortcuts to make sure they can recover the system when networking is down etc.
Everyone and their grandma has access to complex scripts, etc that were once very expensive. Whether they invest the energy in learning methods to hide their presence or just go straight to some other goal is going to depend on how they intend to abuse your systems.
I think most "security professionals" pretend they are going to catch an oddity that occurs that doesn't match what their automated tools would catch and occurs in the middle of some other crisis or holiday break. I would say good luck with that.
That's not true at all. Like, not at all.
You’re assuming the attacker has write access to the log storage of the system they ssh into. This is not the norm for production systems. If your auth processes aren’t shipping off logs immediately, your system is broken regardless of ssh.
If not, can an attacker gain access to your email server or DNS with your ssh key? If either is true, they now have access to everything not protected by 2FA that uses an email address they now control.
There's so many things to get right.
You can design a system such that there's a very high likelyhood even a nation-state attacker won't be able to intrude without leaving traces - if you make no mistakes.
Or you know, you could also just rotate your ssh keys in addition to everything else. "I have logs" is really no excuse to forego something that is this easy.
If the ssh key you lost has access to all the critical infrastructure, then, certainly you have a problem. The solution is to not give away write-access keys to your entire system.
> If either is true, they now have access to everything not protected by 2FA that uses an email address they now control.
The initial question was not whether losing a key would cause a breach, but whether a detection mechanism is reliable.
> Or you know, you could also just rotate your ssh keys in addition to everything else.
The question is how does it help? If you can't detect a breach, it will live for as long as your key rotation policy, if not longer. If you can detect and mitigate a breach, it will be closed quickly regardless of your rotation policy.
In those special cases an "extraordinary" sysadmin gets onboard a log host, it is not through the ordinary access ways, such as SSH from where the other sysadmins play around.
The usual networked logging systems that don't have ssh logins, e.g. syslog into ELK stack.
curl -X "DELETE" http://log-server:9200/logstash-*
And I'm sure I missed some - there's a thousand ways to do it.
This common problem is what led to rise of the MSSPs https://en.wikipedia.org/wiki/Managed_security_service
If you need an ssh key for whatever reason from a host (for example, git pulls on a staging machine), you should generate one on that box and narrow it's scope in the machine that will receive it (eg, Gitlab Deployment Keys -- locked in read-only mode, single purpose).
> The reason rotation isn't recommended is because it leads to bad practices (people just add a '1' to their password)
To some degree. I personally rotate my keys whenever I change my personal/work device (perm. change), or, around every year or so. It's not a hard requirement, but just a personal preference.
> it's a hassle
ssh-keygen -b 4096 isn't a hassle... and if you do deployment properly it shouldn't be an issue to sync these keys (eg, an AD system holding public key, cloud directories like JumpCloud, etc can all sync the moment it's updated -- even Salt/Ansible/Chef/etc can do it easily, with modules pre-written to sync keys already).
In any event, MFA is always a good idea. But, my biggest concern is that someone would leave ssh open to the public... the time it takes to setup an ACL or VPN to connect to the machine is hardly anything these days with the amount of automated tooling to do it... so why aren't people?
I was referring to password rotation specifically with those points, not SSH key rotation, because the quote in my post was also in the context of password rotation.
A lot of people think of the SSH keys like PGP keys (where the one private key is your identity) which is not how they are intended to be used (by my understanding which seems to agree with yours). authorised_keys can contain many so you should never need to duplicate a key because you need to access a given account from multiple hosts.
> That should be a policy and you should write rules to detect when that policy is being violated (check for processes accessing the file).
For static locations, one option I like for this is whitelisting the source address for each key (https://unix.stackexchange.com/questions/353044/). You can then monitor abuse of the policy by looking for keys with no source limit, though this isn't something I've ever done, and it means that a stolen private key is more difficult to use from another location.
Of course this doesn't work for connecting directly from client hosts that move around (i.e. a user connecting from a laptop that could potentially connect from any address unless you enforce VPN access for sensitive resources).
My ssh keys never leave the yubikey.
I have a different dedicated yubikey in each computer, with its own unique key, and a stolen key is useless without its unlock PIN.
Do you leave computers in weird places? Are you getting robbed once a year? Are you a high stakes poker player or CEO of a shady company?
I have like 4 laptops and two main desktops and if I didn't have 4 yubikeys in them then this would be a multiple-times-daily occurrence. Yubikeys aren't that expensive, and I mostly use the usb-c "nano" ones which are designed to live 24/7 in a computer's port, only sticking out about 2-3mm. Sometimes I have to move them around to other temporary machines but for the most part having approximately the same number of keys and computer workstations means that this is pretty infrequent.
I even have two Davinci Resolve Studio activation dongles for this same reason, and I can't physically edit video on two different computers at once, one would do if I were willing to keep track of where it is and shuffle it around between my various machines as needed.
It's pure speed/convenience, not a response to some data threat.
you don't need the physical key to "enroll", you just keep a copy of its pubkey.
So, the ideia is for you to have two devices, each with its own key, the first device you use daily and the second you use store in a safe location.
if your first device in daily use stop working or is lost you use the second device you have stored to login to your systems to remove the keys from the lost device and add the keys for a new device that replace it and then store the second device back in a safe location.
I wrote about my endeavour with this approach just few days ago [1].
[1]: https://github.com/Ciantic/thoughts/blob/master/2021/yubikey...
Most VPS provides "web console" access. It connects like terminal, like Digital Ocean's web-console that doesn't require SSH access [1].
[1]: https://www.digitalocean.com/docs/droplets/resources/console...
As backup you can also use OpenPGP cards which cost much less than a yubikey. Or a cheaper Fido2 token if you use Fido2 for SSH access (I don't yet but it's coming into vogue). An OpenPGP card will cost about a tenner, you'll need a card reader to use it but for backup purposes it's perfect.
Using Yubikey to mean U2F is like people saying "Google this term", "the image is Photoshopped", "Hoover the floor", "grab me a Kleenex", or even "take the escalator". It possible "Yubikey" could become a generic trademark, but if possible people should be wary of using brand names in this way before it 'sticks'.
Additionally, the term Yubikey isn't likely to become synonymous with 2FA in any case. Most people don't know that yubikeys work in several different, independent modes, such as FIDO/U2F 2FA, or CCID smartcard, or Yubico OTP (those long annoying strings your yubikey types when it brushes your thigh or hand).
The CCID smartcard mode requires a pin, which is technically two factor authentication (knowledge of PIN and possession of yubikey), which is an entirely different thing than FIDO/U2F 2FA (which is what most people mean when they talk about using a yubikey for 2FA, not that "yubikey" and "2FA" are interchangeable terms).
This is further complicated by the fact that CCID smartcard mode can be used for ssh (via gpg-agent, with ssh keys inside the yubikey itself), AND, separately, OpenSSH (with other keys) can use a yubikey for U2F.
No endpoint security system I'm aware of being used outside of core banking/telco/government system logs all file accesses. It would crash instantly for one single build of your average NodeJS application.
https://www.newnettechnologies.com/nnt-file-integrity-monito...
It's valuable to hear how companies are protecting their infrastructure if they're going to break down how they accomplished it, how they continue to maintain it, etc.
Also, Sym can be set up to require approvals too which is great for security auditing since it's a third party.
With a properly set-up token, basically you only need 2 commands:
ssh-keygen -t ed25519-sk -O resident -f ~/.ssh/id_mykey_sk
ssh-add -K
Apart from the small headache with the PIN, the whole thing was almost magical in its simplicity.
Edit: I suppose I am asking how does it "all" fit together. The CA that grants servers or services a short lived key - the other servers that then can trust that. It makes kind of sense but I think I am missing some parts as when I try and read how others do it, some parts seem to be missing.
Too many blog posts seem to be something something kubernetes will arrange it. Or Oauth or ...
For example, this article seems to blow against the idea of a central key management service like Vault and have the device decide to rotate keys. But I am not 100% sure because it's one sentence. And how do they provide authentication for a service account (say a web worker that processes some incoming requests). That's not device+person. The same idea can happily apply but do they?
I think I am just moaning.
> check for processes accessing the file
It is. It's called auditd, is quite possibly already installed (albeit probably not configured to do much), and can easily ship its logs off to another host (natively or via syslogd).
It's like, super detectable. You have endpoint logs for the file access, you have network logs, you have sshd logs (which contain the public key and the IP), etc.
> if you have your ssh private key on several machines, you have to remember to copy it to all those places
Your ssh key should never leave a host. That should be a policy and you should write rules to detect when that policy is being violated (check for processes accessing the file).
If you need access from N computers you should be generating N keys.
The reason rotation isn't recommended is because it leads to bad practices (people just add a '1' to their password), it's a hassle, and it can never be fast enough to meaningfully impact an attacker - once they have SSH keys it's likely they can gain persistence and C2 before your rotation takes place. Not because people reuse their passwords in multiple places.
Setting up a CA for SSH is definitely a really good practice but I think that most companies would find it far simpler to just enforce 2FA for SSH access. Still, I'd really like to see an article about how you set that up, especially if it targets smaller enterprise customers.
For others who might be interested, here's bless from Netflix:
https://github.com/Netflix/bless
edit: Oh, and just to be clear, 2FA for your SSH is not a silver bullet - even a yubikey. But it's a cheap, scalable, near-zero overhead way to protect against an attacker who's got access to your key (but not one who has access to your system, assuming an active session!).