It's far more likely that the attacker got legit credentials via another means, web application vulnerability, social engineering or malware attack on company machines, etc. I'd look at the less common applications you run, particularly anything that doesn't particularly look like it was designed to run facing the internet. For example, the elasticsearch guys decided that it would be a great idea to allow anyone who can access it to run java code on the server at one point...
If someone in the datacenter can image the VM and mount it some place in their own machine, reset the ssh rsa key, etc.
Is it good enough to "produce" the proof of the hack?
If so, than no amount of "clean up" can fix the issue, right?
1. firewalling to only the sysadmin's IPs,
2. SSH keys + disabling password logins
6. and disabling SSH on internet facing IPs altogether (if possible).
However it's possible that the attacker's screenshot was of a remote shell initiated via some other means and the OP assumed it was via SSH.
Edit: why was this downvoted? If there's an error then I need to be educated. I've spent enough years of my professional life hardening servers to have some idea what I'm talking about, but I'd be an idiot if I didn't listen to the expertise of others. So please correct me rather than downvote me :)
Probably doesn't help I've been working long hours this week so a little on edge to begin with.
I'm curious how to go from obtaining X info/access to an SSH session.
The rest of that article is very helpful as well!
Foxpass also offers a nice audited, limited duration 2fa ssh auth service.
You're right that key only logins are the easiest quick win, but it's also worth remembering that they can still be vulnerable if the box needs passphraseless keys (eg lazy automation) or if you cannot guarantee the security of those keys (eg they're generated by third parties and/or stored on third party systems). In those situations it would pay to firewall SSH and specifically whitelist the third parties IP address.
Good suggestion about the logs too. That's an often forgotten step yet invaluable when it comes to forensics.
Test the new access before closing the current session! If you made a mistake in any of the steps you will still be able to fix it. If you close the SSH session before testing, and then find that you made a mistake (e.g. forgot to `chmod 600` the private SSH key or something similar) you're stuck!
It is basically zero inconvenience to add an extra argument or shell setting.
However, proper ingress filtering or local iptables/pf rules would stop any unwanted inbound traffic from reaching your server, and you should definitely be using ingress and egress filtering on your network.
If you really want to hide SSH from script kiddies then you're better off setting up firewall rules to whitelist trusted IPs. Heck, even port knocking[1] would be a better option than changing the port number.
FWIW, I've never seen a brute force attack on SSH other than on port 22. Most likely, a targeted attacker would realise if you change the port, you're probably not going to have a trivial password.
Random IPs are unlikely to be port scanned but any reasonably popular site will.
1. firewall - only allow SSH connections from trusted static IPs
2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo, edit /etc/ssh/sshd_config
(edit: make sure the SSH keys have passphrases. That way you have an extra bit of security in case any workstations get compromised)3. Disable root access. edit /etc/ssh/sshd_config
4. Limit SSH access to specific user accounts. This prevents users creating their own key (in the case of mountable home directories) or other machine accounts with passwords (if you've not done #2): 5. Install auto-firewalling for failed SSH logins. I personally favour fail2ban as that covers other scenarios too, but I've also used denyhosts and that's worked well for SSH.6. Lastly, and by far the best option, don't enable SSH on any internet facing IPs.
If you need SFTP enabled, then let me know and I'll post some details on how to harden SFTP so attackers cannot gain an SSH shell.