I think people would be surprised how often it's entirely deliberate. I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops".

I have stopped trying to be the "adult" in our company. I admit that I don't know everything, but two decades of experience has given me some insights. But when I make a suggestion with examples to back them up, my experience is often used against me. I have learned to let it go and allow them to fail on their own. I'm not going to work there for long anyway

So much of this, a company I worked for was like that, I brought up that credit cards were not encrypted/that the monitoring was everywhere but not on critical systems, that dev were using backdoors ( if dev return admin basically) etc, nothing changed. Recently they reported a data leak, no shit...

I think dataleaks aren't punished hard enough. If they did and stung the wallet really hard the industry would quickly adapt to better security requirements.

I'm no so sure -- many managers would just see it as an expeditious cost of doing business, unless it got to kill-the-company-now levels of fines.

I was once talking with an FCC agent about the status of some boards we were importing, and he told me about how Michael Dell would blatantly violate FCC regs with their computers, fight it for a while, then pay the fine out of "Federal Regulatory Account" -- he just happily violated the law and paid the fines enough to have a specific bank acct for that. He was not amused.

I'm pretty sure that it would require a demonstrably real threat of jail all the way up the management chain -- that might focus their minds a bit

No jail, crippling fines that if the behavior doesn't change would wipe them. One example would be sufficient and the industry would fix itself up.

>>unless it got to kill-the-company-now levels of fines.

I did mention that possibility - but to emphasize, it'd have to eb effectively death-penalty for the corporation - just wipe them out.

I'm still not sure that would be sufficient, because there are soooo many examples of horrible managers/executives just torching $$millions and failing up. If any of those executives ever got another job much above the level of sandwich-maker, it would provide no deterrent.

It really looks like at many levels, network and data security needs to be treated as a national security level problem, and any decision to fail to implement state-of-the-art level protections treated as a deliberate compromising of NatSec akin to leaving classified material or CUI (confidential unclassified info) on the sidewalk; not sure what the applicable law is, but that level of negligence is at least the start of serious trouble.

> I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops".

I guess there is some merit to the zero trust security model, since it should urge you to secure every bit of software that you run, without necessarily setting up bastion hosts/DMZs or other types of "server-based ACLs" like that: https://en.wikipedia.org/wiki/Zero_trust_security_model (since i've seen the fact that you have a "perimeter" get used as an excuse to get lazy about security aside from it)

Yet, i've found that it's just too risky, given how brittle most of our software, including its security functionality, actually is. Just the other day i had even the simple blog software that i use (Grav) fail enforcing its admin login functionality, something that seems incredibly simple on the surface and should be bulletproof, yet somehow wasn't: https://blog.kronis.dev/everything%20is%20broken/grav-securi...

Thus, personally i've found that what's worthy of consideration is taking a bit from both approaches. For example, running everything in containers with overlay networks (not even dedicated service meshes necessarily, even the way how Docker Swarm does it is pretty good: https://docs.docker.com/network/network-tutorial-overlay/) that can expose your apps/web servers on every node where necessary, yet keep your DBs constrained to running inside of the overlay network and thus remain inaccessible from the outside.

If and when you need to connect them, maybe run a container with SSH and WireGuard/OpenVPN that let's you forward ports and thus get access to the overlay network, or alternatively just use orchestrator functionality (for example, Kubernetes allows this, albeit i dislike its complexity otherwise) for this.

I failed hard on an edit to pamd, which resulted in any password being accepted for any user. Anything but no password.

Not something I would usually test for, but lesson learned. Thank his noodliness we don't have ssh exposed anywhere outside our network.

Like you, I've got lots of RDS instances (mysql). Generally they're accessible only to certain EC2 instances, so I just SSH into the appropriate EC2 to access the database. There are sometimes cases where it's expedient to expose them, like if I need to quickly test some backend code remotely before deploying it. I keep a security group that allows inbound on 0.0.0.0 :3306 and just assign it if I need to for a couple minutes.

Even if it's only open for a few minutes, there will be logs of malicious actors trying to brute force the DB's password. So how this comes as a surprise to admins is beyond me, but I haven't played with Elasticsearch at all.

Honestly, you'd do better scripting the aws cli to update the sec group with your IP address, which is how we've handled WfH for server SSH/plesk/SQL access.

It's like half a dozen lines of bash, I'll see if I've got it on my home pc.

SSM Port Forwarding and Tailscale are your friend. I'm using a mixture of both with most staff WFH now and their home IP addresses not being static.

Tailscale especially is an amazing product though I've only had it running at work and home for around a month. All that said, a simple script as you pointed out is also great :)

What’s fun is when those IPs remain in the rulesets even after they’re no longer used or needed - and the residential DHCP changes and now someone else has a direct line.

Or it points to an AWS IP that eventually gets handed back and recycled …

And that's where you both implement and act on security policies to mitigate and avoid this based on the threat model you're up against.

That's a cool idea! Never dawned on me to do that.

If you’re already ssh-ing to the database, why not tunnel the db connection over ssh?

It would probably be less effort than updating security rules each time

Ssh tunnels are so badass. I’ve heard of people performing encrypted db replication through them between data centers

I’ve witnessed somebody piping dd over an ssh tunnel to write blocks from one machine directly to another. Highly NOT recommended but still one of the neatest things I’ve seen. It worked.

That sort of thing can be used for making disk images. Restoring them would be the only way to verify them as checking the md5sum of the file is not possible without a reference backup and if you have a reference backup why would you be using dd through a pipe lol. Definitely a cool trick though.

Sounds like a neat party trick! (At least, if you've got a really stable connection.) Did they do an md5sum afterward, or how did they know it worked?

Fair question. I used to do that always. But now, if I'm logged into a bunch of services, AWS, mail, etc in the browser, I don't want to switch my IP address to test something for fear of getting bounced or locked out. Especially if I'm in a rush to fix something.

In that case you'd only have a different IP if you hit the localhost port of that tunneled connection, not everything on your computer. Unless you set up your browser and mail to proxy through that port there wouldn't be any issue with "switching IPs".

This isn't what happens - you do a port forward and you access your services through localhost:port. Nothing else goes through the tunnel. Endpoint services see the ec2 ip. There is no IP "switching" or risk of getting banned.

But if your ec2 service has security group access to multiple things it's extremely useful because you can hit localhost:xxxx for postgres, localhost:yyyy for elasticsearch, etc.

This wouldn't be efficient in my case. I'm testing a frontend that's served on localhost:80 and connects to a local backend also on localhost:80 that makes DB calls to some remote database on 3306. I'd have to modify the backend code to connect through the proxy port every time I wanted to test it. As it is the backend code on my computer is a mirror of the production environment. Normally if I tunnel I have it set up to forward everything through the tunnel except localhost.

In this case you'd simply do the tunnel for 3306:3306.

Your frontend and backend code would run locally and connect to MySQL through the proxy. You wouldn't have to change any backend configuration.

Plus this. I use ssh tunneling with private keys to access my Mariadb or MySQL instances.

I mean, especially on AWS, controlling access to 3306 is dead simple. and literally the one thing that anyone running a DB (which should be everyone) does to ensure that only certain machines, vlans etc can talk to 3306 on said DB instances...

Like literally ONLY this internal IP is allowed to contact the DB.

Also. as far back as 2014 we had AWS scanning all our machines constantly for open ports and alerts on such - we had to get an approval from within AWS to continue to portscan our own machines... they were pretty on top of port scanning behavior...

and I am surprised that the stance they take hasnt improved in all these years.