Cloudflare’s Merkle Town[0] is useful for getting overviews, but I haven’t found an easy way to query CT logs. ct-woodpecker[1] seems promising, too
It uses an API provided by https://crt.sh/
"I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits"
I use LE wildcard certs and they're great, you can use them internally.
Wildcard certs are indeed a valuable tool, but there is no free lunch.
The individual services can still have individual non-wildcard internal-only certs signed by an internal CA. These don't need to touch an external CA or appear in CT logs - only the reverse proxy/proxies should ever hit these, and can be configured to trust the internal CA (only) explicitly.
Renewing a wildcard is also unfun when you have services which require a manual import.
More restrictive is better than less restrictive when it comes to certificates.
Honestly it feels like you'll need at least something like basicauth in front of your stuff from the first minutes it's publicly exposed. Well, either that, or run on your own CA and use self-signed certs (with mTLS) before switching over.
For example, when some software still has initial install/setup screens where you create the admin user, connect to the DB and so on, as opposed to specifying everything initially in the environment variables, config files, or other more specialized secret management solutions.
I wish. I use basicauth to protect all my personal servers, the problem is Safari doesn't appear to store the password! I always have to re-authenticate when I open the page. Sometimes even three seconds later.
I use multiple browsers on my MacBook. Eg I use Google Chrome to run Google Meet, even though I don't use it otherwise.
Just SSH scanning can be a big issue.
You don't need to freak out if you see a bunch of failed ssh auth attempts in your logs. Just turn off password based authentication and rest easy.
And if I'm really, say, in China or Russia an really need to access one of my servers through SSH, I can use a jump host in one of the three countries that I allow.
So effectively: DROPping traffic from 98% of the planet.
Boom.
You want to keep these things behind multiple locked doors, not just one.
For the servers themselves, you shouldn't be able to get to sshd unless you're coming from one of the approved bastion servers.
You shouldn't be able to get to one of the approved bastion servers unless you're coming from one of the approved trusted sources, on the approved user access list, and using your short-lived sshd certificate that was signed through the use of a hardware key.
And all those approved sources should be managed by your corporate IT department, and appropriately locked down by the corporate MDM process.
And you might want to think about whether you should also be required to be on the corporate VPN. Or, to be using comparable technologies to access those approved sources.
What if there's a zero-day in your bastion service, whatever that is?
Few days after they asked to redirect an entire subnet to their rack.
And yes, you still need to remember to close password logins or at least pick serious password if you need them. Helps to have no root login over SSH and normal users that aren't defaults for some distro...
There are plenty of organizations and individuals that can competently run ssh directly on the internet (on port 22) with zero risk to "ssh scanners."
I am just pointing out (also in few other, off-site discussions), that one should not even think of exposing a port before finishing locking it down.
Because sometimes people forget, even experienced people (including myself), and sometimes that's enough (I think someone few weeks ago submitted a story which involved getting pwned through accidentally exposed postgres?).
And there's enough people who get it wrong for various reasons that lowest of low script kiddies can profit buying ready-made extortion kits on chinese forums, getting a single VM with windows to run them, and extort money from gambling/gameserver sites. Not to mention all the fun stuff if you search for open VNC/RDP.
And those people told him the install system didn't support SSH keys (hindsight: they did) and got him to make the root logins possible with passwords. Passwords that weren't particularly hard to guess because their only expected and planned use was for the other team to login for the first time and set their own, before the machines were to be exposed to internet, using BMC.
Not everyone is following that advice. Just last week I taught a friend about using tmux for long-running sessions on their lab's GPU server, and during the conversation it transpired that everyone was always sshing in using the root password. Of course plugging that hole will require everyone from the CTO downward to learn about SSH keys and start using them, so I doubt anything will change without a serious incident.
I had a similar issue 15 years ago, and tied the Linux boxes into Active Directory and authenticated via Kerberos. Worked nice, no SSH keys needed!
A VPN lets me access my stuff from my phone while out of the house, for example.
Original article which doesn't contain the first graphic: https://www.xmodulo.com/access-linux-server-behind-nat-rever...
Tunneling through SSH is significantly worse because you encapsulate a TCP connection inside a TCP connection and it's userspace.
The reason is privacy. I use VPN to obfuscate my IP which means I would have to VPN my entire network. Unfortunately this has proven surprisingly difficult to do properly, meaning with appropriate performance (MTU), IPv6, no blocking (exit IP reputation), etc.
Hence I switched to Argo/cloudflare tunnels for pretty much everything.
I had planned to make the move over the next day, but I moved a single service over to make sure everything was working. Next day as I'm testing moving traffic over, I find that I've been rate limited by Lets Encrypt for a week. I check the database and I had provisioned dozens of certificates for vpn.api-tools.mycompany.com, phpmyadmin.api-tools.mycompany.com, down the list of anything you can think of.
There was no security issue, but it was very annoying that I had to delay the rollout by a week and add a whitelist feature.
What? Ideally..before? Seriously? It is 2024.. and this was true even decades ago, absolutely mandatory.
(Still remembering that dev that discovered file sharing in his exposed mongo instance (yes, that!! :D) without password only hours after putting it up.. "but how could they know the host it is secret!!" :D ).
A good starting point for hardening your servers is CIS Hardening Guides and the relevant scripts.
Once you figured out what the domain is, you can easily build a list of IPs out of the cert transparency log and if there is ever an exploit for this specific type of appliances, attackers now have a bespoke list of IPs to hack, a dream come true.
I don't see a solution for this particular use case, I would argue self signed certs would be more secure in this case.
I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits just like those listed in the article. After a few hours, it always dies down somewhat.
The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.