Ports under 1024 are privileged in Linux - they require root or root delegated privileges to run. When SSH is on port 22, you have the assurance that unless the server is root compromised, it is what you think it is running on that port.
When you make that port 2222 or whatever, like so many people do, you have cut out a lot of noise... but now that compromised PHP application you had running has allowed someone to now race you every time SSH is restarted for an update or crashed or whatever to bind on that port. Let's say it wins - now you've got something listening on the SSH port. If someone using SSH ignores the fact that the host key has changed, now they're trying to login to a fake SSH server. Maybe you use password auth and you just gave them the password. Maybe they're using an OpenSSH client that is vulnerable to leaking private keys in certain situations. Maybe the fake SSH server pretends to be a real shell and they then log whatever actions you try to take when you SSH in. Because they're going to be able to figure out what port SSH is listening on - a fingerprinting port scan can be done in seconds.
You are sacrificing security when it comes to a more focused attacker for the sake of filtering out the low effort mass scans and basic brute forces. The thing is - I'm worried about the former, not the latter.
If you care enough about securing or filtering your SSH to go through any of this trouble, just set up a VPN on a separate machine and restrict SSH access via firewall to that machine. Spin up the smallest VM your cloud provider has, throw up wireguard on it, and you're good to go. It'll be plenty for a VPN that's basically just there for SSH access. Now someone has to have both an exploit for wireguard and an exploit for SSH to get into your machine that has things you care about, you've filtered out all the noise, and you haven't introduced new security risks for a more determined attacker.
It's pretty trivial to leave SSH on port 22 and just forward a high number port to it while blocking 22 externally. All the root-not-compromised assurances and still maintaining the high pass filter.
While a VPN certainly can be a good solution to securing a machine, you've now got the problem of the VPN server needing its access protected.
Public key only for authentication and strict fail2ban rules combined with port forwarding makes for a very tight system. Not invulnerable but secure enough to not be worth the effort.
As I said originally and still maintain, using non-standard ports is a high pass filter. It's not a security measure. It might be part of your security setup but it's just a filter.
All bots will check port 22 first and then move onto others. Your can have a port sniffer watching this and block any attempts to your actual <1024 port of they've also hit port 22.
It works a treat. Shodan, begone. I agree with your high pass sentiment
I think most people saying security by obscurity 1) know that’s a fallacy 2) know that by using that phrase the reader/audience knows what they mean without having to explain a phrase like high pass filter. It’s a bit pedantic to rant on the use of terms.
You could also be exposed to this with key based auth - older openssh clients have a security flaw where they can leak keys, and you could also have a 'fake' shell that still allows them to gain a lot of details about the inner working of your systems.
>It's pretty trivial to leave SSH on port 22 and just forward a high number port to it while blocking 22 externally. All the root-not-compromised assurances and still maintaining the high pass filter.
Yep! Certainly can.
>While a VPN certainly can be a good solution to securing a machine, you've now got the problem of the VPN server needing its access protected.
There's a few things here:
So, Wireguard uses key based auth, so it's pretty sure by default. Additionally, it doesn't show up in port scans - if you don't send the correct key, you get no response from the wireguard server. You'll get the same 'open|filtered' response from nmap and similar that you would for a UDP port with nothing listening. (I don't think this particularly matters - I don't really care if someone knows I've got WireGuard running - I would run it even if you could tell I was. See following paragraph)
Second, if the VPN is compromised (from a key leak) the worst they get is access to whatever the VPN network is. I don't trust clients just because they're on the VPN, and neither should you - they have all the same auth and security requirements I would put in place for something on the public internet. They still need to get past key auth for SSH (or have a zero day for it). If there's a zero day for wireguard itself, well, now they have access to that box, and can potentially use it for nefarious purposes, but I don't keep any private data for my company or our customers on it, so I replace it and upgrade to a version of wireguard without the exploit. They still need to somehow get past SSH auth, or use another zero day. I think the chances of there being simultaneous public zero days for both at the same time is pretty much nil, and someone being willing to burn two private zero days for the two on you means you're up against someone with enough resources you're probably screwed to begin with.
>Public key only for authentication and strict fail2ban rules combined with port forwarding makes for a very tight system. Not invulnerable but secure enough to not be worth the effort.
Frankly, I think public key only auth is most likely enough for 99.999% of everyone to begin with. I don't think fail2ban or port forwarding from a nonstandard port to 22 matters overmuch, even for filtering logs. If you're going for the "realistically good enough" setup, key only, no root, only have personal nonstandard usernames allowed, that log noise doesn't matter because you can just grep for failed attempts for the usernames you care about and ignore everything else, and the only thing you really have to fear is a OpenSSH 0day, which running on a nonstandard port isn't gonna save you from either. It might buy you a little time to get it patched and not get hit in the first wave, but that's about it.
It's like putting your key under your door mat vs. some loose brick near your back door. Sure, someone can still find it (I don't recommend the real-life equivalent), but assuming you have a camera and/or nosey neighbors, there's a good chance the presumed invader is going to look suspicious enough to garner unwanted attention.
Especially when it involves IPv6 networks and each internal device has its own IP, this can involve a lot of scanning. I haven't had any complaints but good point, I could get banned for that.
Here's the real problem.
Also if you're worried you could use SELinux, AppArmor, or things like docker to limit the capabilities of a compromised PHP application.
You should definitely use SELinux or AppArmor. For any production workload I do.
But for any production workload I don't allow SSH over the public internet. I require a VPN connection to a subnet that only has jumphost(s), the jumphost access logs are shipped to an ELK stack, SSH requires both a key + MFA to login to the jumphost, and SSH between production servers is blocked and alarmed on.
The idea that you never ever have to SSH into a production server is a nice ideal, but I've never seen it survive reality unless you just shrug about issues occurring and don't mind not being able to root cause them.
Then it's game over regardless of anything else. That someone is now owned by anything and anyone who cares to MITM them.
But it's also a world where people have grown accustomed to doing this - automatic scaling in the cloud with re-use of IPs means that systems come up with the same IP frequently, so people have gotten used to doing this.
For the stuff I manage in the cloud I use their APIs to keep my known_hosts file up to date - if an IP got reused by an instance, I clear the entry in my known_hosts file. If it didn't, I would see the key change alert (verified by modifying the host key on the destination server)
Wireguard doesn't answer anything during port scanning, or when it receive unauthenticated packets, as it doesn't need a handshake.
Actually, I'd think that wireguard + telnet should be good enough, though ssh has a lot more features.
While I agree that UDP port scanning is harder than TCP, since you cannot just batch send SYNs on every port, it really depends on how you did setup iptables and the sophistication of the port scan.
Most scanners will consider that not receiving an ICMP/port unreachable is a sign that some UDP service is listening. This could be prevented by a default DROP instead of REJECT to confuse scanners, but it has other annoying implications. I bet most people out there do use REJECT so a UDP service such as wireguard would be immediately spotted because of its lack of response.
Also note that if the attacker is willing to invest a bit more time on the map, they will most likely have a Wireguard probe.
Most real world firewall configurations I have seen use DROP instead of REJECT, so anecdotally I'm not sure about your claim.
I don't think it's particularly worrisome if someone knows I'm running Wireguard, though.
If there's a wireguard zero day, they gain no more access to the rest of my servers than if I had port 22 open to the internet - and I don't think that's a significant risk with key based auth to begin with. If there's an openssh zero day, then yeah that could be trouble, but now someone needs to have zero days for both wireguard and openssh to get in to my prod servers?
If they've got two private zero days for that and they're willing to burn them on me, I'm an incredibly high value target and pretty much everything we're discussing here isn't going to be enough to save me, and I've got much bigger problems.
Is that really so? I'm not all that familiar with Wireguard but that seems like a debugging nightmare if the client has no way to get any error pointer from the server. At least my experience from setting up some IPsec infrastructures is that client logs are essential for troubleshooting.
> Most real world firewall configurations I have seen use DROP instead of REJECT, so anecdotally I'm not sure about your claim.
I do advocate for DROP, but most configurations I see are default REJECT. DROP has a bunch of disadvantages that most people don't want to deal with. It messes with TCP because of the lack of ICMP responses, and overall it makes troubleshooting harder because you end up with programs hanging and time outing instead of instantaneous failure.
Actually the use of REJECT is so widespread that it makes my life easier. When multiple levels of firewalls are involved, I can be quite sure that if the program hangs, it's a rule on my side, while a RST tells me that it's somewhere else.
> I don't think it's particularly worrisome if someone knows I'm running Wireguard, though.
Agree, though the subject is just port scanning here, not what happens beyond that.
Yep! Which, you're right, can make wireguard troubleshooting a pain... but it also mostly 'just works' with significantly less configuration overhead and chance of messing things up than your general IPSec IKE setup. IKEv2 definitely makes things nicer, but it's still not as generally painless as wireguard.
I don't know why to mention them in the firewall at all. The kernel knows which ports it is listening on, and drops the rest without prompting.
Because 99.99% of people out there use iptables, with a single default rule for unmatched packets, whether it be UDP or TCP.
> I don't know why to mention them in the firewall at all.
Because you surely don't want your hosts to expose something unexpected to the internet .
> The kernel knows which ports it is listening on, and drops the rest without prompting.
Nope, and that's precisely the subject of the discussion here. In case nothing is listening, you will receive a nice ICMP "port unavailable".
Is that the same (externally seen) effect as a REJECT?
I don't think it's particularly likely, but I don't think OpenSSH server zero days are particularly likely either.
This attack is mitigated quite strongly if you have selinux enforcing and aren't running networked services in unconfined domains.
Even if you have networked services that aren't confined in the default targeted policy, you can probably learn to write policy for them in a day or two, although poor documentation can make for a steep learning curve.
This is what's weird to me about this whole argument - people can come up with lots of ways to secure this, yet aren't willing to do one of the things that will provide the most security while also offering a high pass filter in blocking SSH access to the public internet. No log noise, no chance of a zero day hitting it, one key being compromised is no longer enough to result in an access breach.
Adding a dedicated bastion would double my monthly costs, but SELinux costs me nothing if the targeted policy covers my applications, or like half an hour of my time per service if I have to write my own policy modules.
Although, I should point out I'm playing devil's advocate here because my ssh is still on port 22.
So create a tunnel to create a tunnel? Why not port knocking or just white listing static IPs?
I'm not actually joking -- the above tricks work really well for stable connections through the China firewall.
Because the public facing SSH services I have on IPv6 have never been probed or had bad login attempts even once in the last four years. I don't consider that a security control at all.
SSH is bit tricky though, because you may find yourself on a laptop in a hotel and then you would first need to create an IPv6 tunnel to be able to get in. If the only time you need to get in is during an emergency that's a hassle you don't need.
It's a pity IPSEC is still such a total dumpster fire on Windows as it works great for ad-hoc with everything else, but thankfully Wireguard has been about as light-touch as VPNs get and I've had good luck getting it out of nearly any network.
If you're doing this for security reasons, then it is by definition 'security through obscurity'. The problem is that you don't like the term because it generally carries a negative connotation. What you and the author should be arguing is that 'security through obscurity' shouldn't be seen as a negative necessarily.
>So when you have SSH listening on some uncommon port and see hits in your logs you have a reasonable assurance the attacker isn't some script kiddy spamming the "attack" button on some stupid app.
How do you figure that?
Well the high pass filter's purpose is to increase security. Therefore it is literally security through obscurity... Even if it's effective, that's what it is...
Anything that reduces the likelihood of success of any attack is security. Non-standard ports essentially remove you from the radar for a lot of dumb IP block scanners. That alone is worth the minimal effort involved.
It won't be inherently more secure, the same flaws will still exist, it will just be less visible to attackers, thus obscurity.
Hiding things has its value. Learn how to use all tools at your disposal, wisely.
One could argue all security is "through obscurity". The encryption key exists somewhere in the parameter space, which is open to everyone. You just don't know where it is.
It is more productive to think of the onion model of security. Any technique is just another layer in the onion. As long as you have a sufficient number of layers, and they're all good in different ways, you're probably fine.
Of course not, and frankly that's a gross mischaracterization of the parent's point. The problem is not that security through obscurity is the devil; the problem is that security through obscurity is not security, and is not useful when you need security.
> The encryption key exists somewhere in the parameter space
This misses the point of cryptography, which is that you (roughly) have to try each possible key separately. So, for example, the combination of a 64-bit security layer (or set of layers) with another 64-bit security layer is a 64-bit security layer, not a 128-bit security layer, which you should know if you know what "parameter space" means.
Log filtering has about as much impact on security as having overseas contractors watch your security cameras.
Ways I've seen this go wrong:
- someone fails to copy the port while communicating it
- the new whiz-bang AI security solution detects non-standard SSH ports and "quarantines" you while you try to figure out what happened and who to talk to
- someone manually "reviews" the firewall rules and locks you out of your own boxes
- someone builds a tool that uses SSH but doesn't allow non-standard ports
> someone manually "reviews" the firewall rules
This is a feature. Allowing Unfriendly AIs or incompetent morons to dictate security policy will go horribly wrong eventually; nonstandard SSH ports help make it painful for you to allow them to gain a foothold in the first place.
I have certain hosts behind a single IP and forward SSH to them on arbitrarily chosen ports.
Do I still get random logins on those ports?
Why yes I do.
Does putting SSH on a different port make any difference?
No it doesn't.
Putting SSH on a different port is either done for a specific reason, or you're just deluding yourself that you've somehow reduced your attack surface.
On the other 20+ boxen with ssh on port xxx22 the logs and f2b rules are much smaller - which means less hassle for the admin.
And on the boxen with services behind WG there is zero noise.
Naturally we're using keys only with all this. The reduced noise in the logs/rules/firewall are very handy.
[0] https://www.youtube.com/watch?v=xkrMsPiqG6M&feature=youtu.be...
Yes. The specific reason is "it makes log files less full of crap we have to sift through".
> So when you have SSH listening on some uncommon port and see hits in your logs you have a reasonable assurance the attacker isn't some script kiddy spamming the "attack" button on some stupid app.
Do you remove a vulnerability by doing this? Whatever you are afraid that the script kiddy may do, still can happen, thus it is security by obscurity.
> You also have reasonable assurance blocking that host/network with fail2ban or equivalent will actually be effective.
Can you explain that to me? How is it less effective to block a script kiddy or not? Block him... that's it...
> You can also set way more strict fail2ban rules since you're far less likely to ever lock out authorized traffic.
Wait, what? How will it make less likely to block authorized traffic? As long as that traffic act like authorized traffic, it should still be authorized, isn't it? You would blanked ban traffic that doesn't try too many username/password?
Reducing the probability of the attack attemp is good in itself and reduces the noise and thus allows easier detection of worse attackers in server logs.
>thus allows easier detection of worse attackers in server logs
I would be amazed if you could show me a single instance where people would see an attacker in server logs and act on it... quickly enough too that it's effective.
I'm pretty sure that it's extremely rare that anyone will look at logs, even small one, to catch theses kinds of situation. If that's really what you suggest as being an effective way to protect yourself, than that's what you should mentions that it's required to do to become effective.
Fail2ban will act noise or not... which was my point in my comment.
On my little jump server at home I used to get tens to hundreds of login attempts per hour. The server only allows for pubkey authentication and my fail2ban rules were strict (block after a single failed login) so the probing never went anywhere. Unfortunately the block lists got huge if I set the block time to a meaningfully long period. It also kept my server at a constant low utilization.
I then started forwarding a non-standard port (that didn't end in 22) and the number of attempted logins dropped to a dozen in a week. I could set my f2b rules to expire after days and not see any performance impact from a big iptables blocklist. My server's baseline load also dropped since it wasn't dealing with constant noise. I could also set up alerts on failed login attempts since I wasn't getting inundated with noise.
I started with a secure setup and increased security by changing the SSH port. Less noise means I can actually see when things happen. It's an outlier event when I get a failed login on my server.
Why would you let the rule be shorter when the attack are more targeted? Doesn't make sense to me to lower it.
> not see any performance impact from a big iptables blocklist.
Wouldn't it be a pretty marginal performance impact? Let say you do get a performance impact significant, then the goal isn't security, it's performance. That's a new thing, performance by obscurity!
> I could also set up alerts on failed login attempts since I wasn't getting inundated with noise.
So you are looking for failed login attempts more now? I have an hard time believing this, but as I said in my previous message, if that was what made it more secure, it's a pretty important step to add, as I would ignore them whether it's 1000 an hour or 10 a week.
I'm curious, what would you even do if you were to see a failed attempt? At what point will you act and how would you act? Anything I can think of is meaningless.
I set my bantime and findtime to much longer durations and maxretries to be 2. This means my iptables block list is long for long periods and f2b has to do more work looking through the SSH logs when dealing with the noise of listening on port 22. Both have non-zero effect on my little Atom powered jump server.
Cutting down the noise let me set even stricter f2b rules with far fewer resources being used on my server.
As for "doing something", there's not much to do for a failed SSH login attempt. I do send myself notification emails but those are just a small section of a larger status email. If I were to see a lot of login attempts on my random high port I'd likely take down external access to my server or set up VPN access to my home network.
My goal is to avoid background noise and get on a "not worth it don't bother" list.
There's billions of hosts on the Internet and many millions of them running an SSH service. Even attackers with huge botnets at their disposal have finite resources and time to probe for hosts and then try exploiting vulnerable ones. Every exploit attempt has some percentage chance of being against a honeypot used to help catch the attacker.
There's little point in even sniffing non-standard/uncommon ports for different services. Anyone changing a service port has a higher likelihood of having a more secure system than someone with standard ports. So even if you scan high number ports for an SSH service you're unlikely to find one on a high port that's not using a more secure than default configuration.
So when you have SSH listening on some uncommon port and see hits in your logs you have a reasonable assurance the attacker isn't some script kiddy spamming the "attack" button on some stupid app. You also have reasonable assurance blocking that host/network with fail2ban or equivalent will actually be effective. You can also set way more strict fail2ban rules since you're far less likely to ever lock out authorized traffic. The more specific your allow criteria the better and more effective your block rules can be.
A "Beware of Dog" sign doesn't physically secure your house on its own. You might not even have a dog. What it does in change the calculus for a burglar. Why risk it when there's more obviously less risky targets nearby? The sign is a high pass filter.