> Using a non-standard SSH port is a bad example because nmap can see through that deception in a few seconds.

Compared to milliseconds. Do yourself the favor and open one sshd on port 22 vs one on a port >10000, then compare logs after a month. The 22 one will have thousands of attempts; the other one hardly tens if even any.

The 99% level we're defending against here is root:123456 or pi:raspberry on port 22. Which is dead easy to scan the whole IPv4 space for. 65K ports per host though? That's taking time and, given the obvious success rate of the former, is not worth it.

Therefore I'd say it's the perfect example: It's hardly any effort, for neither attacker nor defender, and yet works perfectly fine for nearly all cases you'll ever encounter.

EDIT: Note that it comes with other trade-offs, though, as pointed out here: https://news.ycombinator.com/item?id=24445678

I know we've spoken in another thread, but I think it's important for people to understand that this sshd thing is a perfect example of why it isn't this easy: You reduce log spam moving to a non-privileged port, but also reduce overall security - a non-privileged user can bind to a port above 10k, but can't bind to 22. sshd restarts for an upgrade, or your iptables rules remapping a high port to 22 get flushed, that non-privileged user that got access via a RCE on your web application can now set up their own fake sshd and listen in to whatever you are sending if it manages to bind to that port first and you ignore the host key mismatch error on the client side.

Or you can implement real security, like not allowing SSH access via the public internet at all and not have to make this trade off.

Here's a counter-example (I said else-where in this thread):

Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22.

I'll also point out that we're generally talking about different threat vectors here, so it's good to lay them out. I don't think obscurity helps against a persistent threat probing your network, it helps against swarms.

> a non-privileged user can bind to a port above 10k, but can't bind to 22. sshd restarts for an upgrade, or your iptables rules remapping a high port to 22 get flushed, that non-privileged user that got access via a RCE on your web application can now set up their own fake sshd and listen in to whatever you are sending if it manages to bind to that port first and you ignore the host key mismatch error on the client side.

This is getting closer to APT territory, but I'll bite. If someone has RCE on your SSH server it honestly doesn't matter what port you're running on. They already have the server. You're completely right it would work if you have separate linux users for SSH and web server. Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). But let's assume it here. In reality, even if you did have this setup this is a skilled persistent threat we're talking about (not quite an APT, but definitely a PT). They already own your website. Your compromised web/SSH server is being monitored by a skilled hacker, it's inevitable they'll escalate privileges. If they're smart enough to put in fake SSH daemons, they're smart enough to figure something else out. Is your server perfectly patched? Has anyone in your organization re-used passwords on your website and gmail?

You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact:

* Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!). * Use standard port, but you still have an APT who owns your web server and will find other exploits.

>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22.

Yep! And I should be clear: I am not saying just don't change the SSH port. I'm saying if you care about security, at a minimum disallow public access to SSH and set up a VPN at a minimum.

>Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them).

I'm a bit confused here. In every major distro I've worked on (RHEL/Cent, Ubuntu, Debian, SUSE) the default httpd and nginx packages are all configured to use their own user for the running service. I haven't seen a system where httpd or nginx are running as root in over a decade.

I think the bare minimum for anyone that is running a business or keeping customer/end user data should be the following:

1) Only allow public access to the public facing services. All other ports should be firewalled off or not listening at all on the public interface

2) Public facing services should not be running as root (I'm terrified that you've not seen this to be the case in the majority of places!)

3) Access to the secure side should only be available via VPN.

4) SSH is only available via key access and not password.

5) 2FA is required

I think the following are also good practices to follow and are not inherently high complexity with the tooling we have available today:

1) SSH access from the VPN is only allowed to jumpboxes

2) These jumpboxes are recycled on a frequent basis from a known good image

3) There is auditing in place for all SSH access to these jumpboxes

4) SSH between production hosts (e.g. webserver 1 to appserver 1 or webserver 2) is disabled and will result in an alarm

With the first set, you take care of the overwhelming majority of both swarms and persistent threats. The second set will take care of basically everyone except an APT. The first set you can roll out in an afternoon.

With the first set, you take care of the overwhelming majority of situations.

Protecting sshd behind a VPN just moves your 0day risk from sshd to the VPN server.

Choosing between exposing sshd or a VPN server is just a bet on which of these services is most at risk of a 0day.

If you need to defend against 0days then you need to do things like leveraging AppArmor/Selinux, complex port knocking, and/or restricting VPN/SSH access only to whitelisted IP blocks.

Except you don't assume that just because someone is on the VPN you're secure.

If the VPN server has a 0day, they now have... only as much access as they had before when things were public facing. You still need there to be a simultaneous sshd 0day.

I'll take my chances on there being a 0day for wireguard at the same time there's a 0day for sshd.

(I do also use selinux and think that you should for reasons far beyond just ssh security)

A remote code execution 0day in your VPN server doesn't give an attacker an unauthorized VPN connection, it gives them remote code execution inside the VPN server process, which gives the attacker whatever access rights the VPN server has on the host. At this point, connecting to sshd is irrelevant.

Worse, since Wireguard runs in kernel space, if there's an RCE 0day in Wireguard, an attacker would be able to execute hostile code within the kernel.

One remote code exploit in a public-facing service is all it takes for an attacker to get a foothold.

I do not run my VPNs on the same systems I am running other services on, so an RCE at most compromises the VPN concentrator and does not inherently give them access to other systems. Access to SSH on production systems is only available through a jumphost which has auditing of all logins sent to another system, and requires 2FA. There are some other services accessible via VPN, but those also require auth and 2FA.

If you are running them all on the same system, then yes, that is a risk.

For a non-expert individual who would like to replace commercial cloud storage with a self hosted server such as a NAS, do all these steps apply equally?

I am limiting the services to simple storage.

Looks like maintaining a secure self cloud requires knowledge, effort and continuous monitoring and vigilance.

Most of those are good practices for a substantial cloud of servers that are already expected to have sophisticated configuration management. They're easy to set up in that situation, and a good idea too because large clouds of servers are an attractive target - they may be expected to have lots of private data that an attacker might want to steal and lots of resources to be exploited.

A single server run by an individual and serving minimal traffic would have different requirements. It's a much less attractive target, and much harder to do most of those things. For example, it's always easy and a good idea to run SSH with root login and password authentication disabled, run services on non-root accounts with minimum required permissions, and not allow things to listen on public interfaces that shouldn't be. Setting up VPNs, jumpboxes, 2FA, etc is kind of pointless on that kind of setup.

>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22.

But how much of a threat is this? Who's going to drop a ssh 0day with PoC for script kiddies to use? If it's a bad guy he's going to sell it on the black market for $$$. If it's a bad guy he's going to responsibly disclose.

>You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact:

>* Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!).

But blocking 50% of the hacking attempts don't make you 50% more secure, or even 1% more secure. You're blocking 50% of the bottom of the barrel when it comes to effort, so having a reasonably secure password (ie. not on a wordlist) or using public key authentication would already stop them.

It makes the logs less noisy. And with much less noisy logs it is easier to notice if something undesirable is happening. Also from my experience this 50% is more like 99%.

> Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them).

If you made a list of things like this which annoy you, I would enjoy reading it.

> Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22.

And with all those compromised servers they could easily scan for sshd on all ports.

Yeah, I'm seeing a lot of nonsense in here. Why is SSH publicly accessible in the first place???

Security through obscurity is just some feel good bullshit.

Why shouldn't SSH be public? It is useful and simple to secure.

Well, there's basically two stances you can reasonably take:

1) SSH is secure enough just by using key based auth to not worry about it.

2) SSH isn't secure enough just by using key based auth so we need to do more stuff.

If you believe #1, then you don't need to do anything else. If you believe #2, then you should be doing the things that provide the most effective security.

Personally, I believe #1 is probably correct, but when it comes to any system that contains data for users other than myself, or for anything related to a company, I should not make that bet and should instead follow #2 and implement proper security for that eventuality.

I'm willing to risk my own shit when it comes to #1, but not other people's.

Fair enough, I've edited the comment to reflect this :)

> The 22 one will have thousands of attempts

The range in the figures is surprising. I leave everything on port 22, except at home where due to NAT one system is on port 21.

On these systems, since 1 September:

  lastb | grep Sep\  | wc -l

  160,000 requests  (academic IP range 1),
  120,000 requests  (academic IP range 2),
    1,500 requests¹ (academic IP range 3),
    1,700 requests² (academic IP range 3),
  180,000 requests³ (academic IP range 3, just the next IP),
   80,000 requests  (home broadband),
   14,000 requests  (home broadband ­— port 21),
    5,000 requests  (different home broadband, IPv4 port)
        0 requests  (           ,,     ,,      IPv6 port)

¹²³ is odd. All three run webservers, ² also runs a mailserver, yet they have sequential IP addresses.

I don't bother with port knocking or non-standard ports to ensure I have access from everywhere, to avoid additional configuration, and because I don't really see the point when an SSH key is required (password access is disabled).

Good example, but doesn't help his point, which was:

> This has it completely backwards. Security through obscurity's goal is not to signal predators, it's the opposite. The goal is to obscure, to hide. The "signal" is there is nothing here

An attacker scanning the whole IPv4 space won't think "ah, there's no ssh on port 22, there's no ssh to attack". They will think "yep, they did at least the bare minimum to secure their server, let's move on to easier targets".

He proved the point he was trying to disprove.

So I get 10s of attempts a day for my sshd on port 7xxx. If I had an account with say ubuntu:ubuntu, it'd totally have been found by now.

I have 0 in the last 14 days on port 2xxx. Probably depends a lot on your IP range (I'd assume AWS etc is scanned more thoroughly) and whether you've happened to hit a port used by another service. But even in commercial ranges, I've seen hardly any hits on >10k.

But I have only anecdotal evidence as well, so my guess is as good as yours.

So scale the other aide up too. Just imagine what it's like on 22.

The article addresses this. He did a poll, and just under 50% of people use the default ports. So just by changing your default port, you eliminate half the break-in attempts.

Now you're absolutely right that this only deters less-skilled/inept hackers, a more competent hacker easily gets past this. But it's worth dwelling on the fact that we still stopped a substantial number of requests. Port knocking is definitely an improvement (i.e. more obscure). I'd guess with port-knocking more than 90% (even 99%) of attempts would completely miss it. The goal here isn't to rely completely on obscurity. It's security in depth. Your SSH server should still be secure and locked down.

The other question with this is what's your threat vector. Most people decry security through obscurity because an APT can easily bypass it. They can, but most people trying to hack you are script kiddies. Imagine an SSH exploit was leaked in the wild – all the script kiddies would be hammering everything on port 22 immediately.

The poll is my biggest issue with an otherwise agreeable article, the sample size and representation on Twitter doesn't make for anything close to reliable percentages.

I understand its use as a demonstrative aid but especially in the context of security, hinging your policies on the outcome of a Twitter poll seems like... well, security through obscurity.

Maybe a bit nitpicky but I think port-knocking is in kind of a grey area. You can think of it as a kind of password where you have to know the correct series of ports. Since the number of ports is quite large, there is also a correspondingly large number of possible port sequences so you can't, in principle, brute force it without a lot of effort.

> Maybe a bit nitpicky but I think port-knocking is in kind of a grey area. You can think of it as a kind of password where you have to know the correct series of ports.

Yes.

But you also have to know that port knocking is enabled at all. That's the obscurity part.

Eh, I think its how others have expressed in this thread.

Security by obscurity is bad, not because it is bad to have well secured countermeasures, but because it encourages poor thinking with regards to methods you have in place, and additionally because it usually introduces extra, unintended attack vectors.

You suffer from your own 'obscurity' - whether its because you forgot you had to port knock, use a different port, or because you somehow managed to leave a new exploit in some obscured code due a bug in the obscured code, or because you managed to open yourself up to an RCE with the port knocking, or some other obscure scenario you did not intend to create from whatever obscurity you created.

I think this is different from Defense in Depth, which just says have more than one countermeasure in place, and to keep the counter measures 'separate', but well defined. Port knocking, but on a different box than your vpn box, on a different box than your ssh box.

We aren't told to use 'well-defined' passwords, like 1234, obviously. If the point of 'obscurity' is 'secrets', that's all well and good, but that's not security, that's a password. Have a password, by all means, but tunnel it over TLS and use well defined security paths versus creating unnecessary risks.

I think this implementation avoids that problem.

https://github.com/moxie0/knockknock

Gotcha. Fair point.

Less secure than a password because anyone sitting in the middle can observe the sequence.

> [port-knocking] is very much security by obscurity, because it's trivial for anyone who knows the port sequence to defeat

in what way is this different than a passphrase you don't know? i can trivially defeat any password which i already know, too :D

while discovering a non-standard ssh port is easy, discovering a port-knock sequence out of a possible ~65k per knock is impractically difficult (assuming the server has any kind of minimal rate limiting). a sequence of eight knocks will need 65k^8 attempts - and that's assuming you already know which port will be opened, which of course you won't.

you can even rely on just port-knocking of 8 ports and already get ~2^48 bits of entropy, which is about the same strength as a random 8 char alpha-numeric latin-charset password.

(someone plz check my math)

I agree with you that the example is not the best, but obscurity has a lot of benefits. We did an experiment with a few students on obscuring a WordPress installation some years ago to catch ppl scanning for certain plugins. That gave us the ability to use the regular paths as honeypots. Gives you an ability to detect 0-day attacks as well.

Changing the ssh port would still fall under security through obscurity whether its effective or not.

This doesn't consider that you can then monitor connections to port 22, and if you see any that's suspicious.

Disagree that port knocking is obscurity. That's a secret.

Security through obscurity would be using a nonstandard SSHD service.