Just want to yes and you.

In general, put everything in private subnets, and make the only way any traffic can get to a server is through a load balancer. There are very few reasons to have a server itself have its own public IP address, and using your load balancer as a chokepoint, means you can set up layers and layers of redundancy to prevent traffic from ever being able to reach a database under your control.

This holds true whether we're talking RDS DBs, something you've spun in a K8 cluster, or something you're running on a vanilla compute box.

Assume you will screw up at some point and open up a port that shouldn't be. Ask yourself, "what's the impact here?" If you're OK, even with a fat fingered port, great.

Assume you will have a dev deploy a DB without a password. "What's the impact here?" If you're ok, even without a password, great!

That isn't to say that the above two scenarios are something you should tolerate, but automation can help detect these sorts of issues and make it easy for you to resolve them. While that automation is running, you want to make sure you're not going to get owned.

Kudos to the OP for sharing their story. Always lots to learn on this front, and we can always get better at our cloud operations.

Defense in depth... VNets, TLS certs, RBAC etc.

Layer 4 and/or 7 proxies -> private networks -> services. Auth and encryption all the way on everything. IP white-listening were suitable. Pretty much the standard way of building web were I live.

"Private subnet" is just one way to accomplish it - what you want is a firewall that drops inbound traffic by default, and private subnet/NAT is one common way to do it that does have some downsides (extra complexity - you need a NAT gateway now).

+1 on the private subnet.

You want to design your system so that if a network a critical misconfiguration occurs you don't open yourself up - you simply stop working.

(Too many years chasing EMR clusters getting dropped onto the internet by users with AWS console access).

I agree with only the third of those. The other lessons I'd take would be:

1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners

2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)

Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs.

As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ...

I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.

All internal services should be protected, even on your home network.

If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.

At every layer of “breakage”, you are cutting down the chance for an attacker to access your system. While it’s possible the failures might cascade it seems likely that these failure modes are pretty independent of each other.

You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.

I think you misunderstood me - I mean that you want multiple layers to your security, and you need to be careful to select layers that fail independently. If Vault has a sufficiently bad unauthenticated bug, then attackers can simply use it to request database credentials and query the DB, which is now on the open internet.

An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."

My view is that for most cases the cost/benefit of multiple layers doesn't stack up. Given a fixed amount of available time and effort, you'll generally get better results by focusing that effort on making one really good layer - e.g. putting active monitoring in place so that you detect when your single layer breaks (whether that's an attacker from outside the network being able to connect to inside the network or an attacker with an expired certificate being able to connect to a live system) - than by spending half as much time each on two layers, IME.

Unless this is literally running on a single 1U host in some colo, there is no excuse for not having defense in depth. An old school DMZ if you are in a datacenter. A VPC if you are on the cloud. Then client certs for everything. Two factor for ssh. Auditing. These are straightforward to set up, with different options if you have time-but-not-money or money-but-not-time. If you have not-money-and-not-time then this is probably going to end badly no matter what.

> there is no excuse for not having defense in depth. An old school DMZ if you are in a datacenter. A VPC if you are on the cloud. Then client certs for everything. Two factor for ssh. Auditing.

Maybe there is no excuse, but literally every company I've worked for (including Fortune 500s) has been missing at least one item from your list. So "industry best practice" means committing less time and money than it would take to implement all those things properly (rightly or wrongly) and you need to triage and prioritise.

So the excuse is "my company doesn't take infosec seriously". Like every similar issue of "should do but don't" (testing, formal promotion processes, diversity, harassment response), you get to decide to tolerate it or get a job elsewhere. My experience is that the companies that take this stuff seriously also do a better job of converting my skills into cash, and as a result, pay me better.

Interesting. My experience is that companies that "move fast and break things" have been better at making money and paying me (whereas I personally lean too far in the perfectionist direction). Interesting businesses face a wide range of risks - competition, regulatory, market - and infosec is rarely the biggest one IME.

I just want to say that I think this is absolutely a fair approach for many systems, but it sounds so radical against the backdrop of perimeter security that you're catching undue flak. You need to take a lot of care to build and design the "one layer to rule them all" in a way that's ridiculously sound, but this can actually produce a strong design if you pull it off. I didn't give this advice because I looked at where NewsBlur is today and figured they had some learning curves to get over before they could make the right tradeoffs to do it safely. If they got that much wrong with their DB, they might not know how to design a sane zero-trust network, or even how to make rational decisions in that world.

You see this "authN/authZ above all else" line of thinking in Google's security design [0], with their ubiquitous login wall. For their employees, that login wall has extra hardening - you can't do regular password resets, and you need to possess a physical security key (which acts basically as a scaled-down single-purpose HSM) and pass a suite of posture checks on the device (proportional to the sensitivity of the protected resource) to pass through it.

Then they put this login wall in front of everything, even internal services, and that tends to be OK because the system "fails closed," and the only way to access their protected resources is via physical safe rooms deep in the Google offices in which elevated privileges may be obtained.

Putting all your systems on the Internet forces you to get the incentive structure right, and that can be useful in companies where "private networks" can serve as justification to weaken security - if your login page is Internet-exposed, then you simply have no choice but to make it strong enough to withstand the chaos of the Internet. There's significant merit in that.

[0]: https://sre.google/books/building-secure-reliable-systems/

> 2. Use real authentication rather than network firewalling.

This is in vogue but it’s wrong because it presents this as an either/or.

In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone).

There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges.

People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.

In reality time and attention are limited resources. Does network-level filtering have a good RoI relative to putting the same effort into improving security at the service level? IMO no (at least not until you've reached a very high level of service security where you're hitting severely diminishing returns) - network-level security is necessarily at least one of a) crude b) complex enough that it becomes an attack surface itself - though reasonable people can disagree.

You are looking it as a network level filtering.

I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it.

In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public.

This is easily, done on something like AWS.

The very concept of a "private network" leads you down the path of making a security boundary that has far too large a surface area, IME. Either it becomes a big bag with all your hosts in it, and you write systems that trust all the requests they get (even if you know you "shouldn't") because you know only your systems are on the network, and then an attacker figures out how to make one of your systems make an arbitrary request and you get owned. Or you put each component on its own private network but then you have to open up every port you're actually listening on so that your components can talk to each other (and you probably automate that in your kubernetes/puppet/whatever setup, like what happened here with docker) and the private network does nothing.

An independent firewall accomplishes the same even if every server has a public IP.

Respectfully, all of the suggestions are good, as are your two. But, segregating your public traffic and private traffic is an excellent way to prevent these sorts of issues from happening. While it doesn't prevent someone who's compromised your network from accessing the DB, it means someone can't fat finger a port and open a DB to the public internet.

Well, your second point is his first point.

And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.

What strikes me about your reply is that 1, 2, 3 are all absolutely basic table-stakes things. SQL injection mitigation level things.

Have we suddenly stopped teaching the basics?

Youre downvoted but you’re right - this is entry level stuff and it seems to get routinely not taught or simply forgotten and ignored.

Everyone seems to want to worry about “nation state actors” and getting with novel 0 days when in fact missing the basic low hanging fruit is likely to result in far more damage

I'd also suggest to use another server/vps to monitor your setup (availability, performance, ...), and include a ports open check in the monitoring tools. Having an external check of your usual ports (thoose that should be closed) is a good way to find when something is off.

Another takeaway, IMO, use infrastructure-as-code to define the components in your infrastructure, so something like this critical firewall configuration won't be missed

Docker will happily override your firewall rules even with infrastructure as code. We block the AWS/GCP metadata IP to avoid potential exploits using that to pivot, but Docker also manages to override that sometimes, so we have a cron job that re-applies it.

Another important missing point is not having docker publicly exposed. Just reach docker through a ssh tunnel (eg via rdocker) or a vpn.

Can I also add don't run MongoDB into Docker containers. Use MongoDB Atlas or switch to AWS and DocumentDB with proper VPC and a network level firewall (security groups).

@conesus It's just not worth using discount hosting providers for this exact reason. Use a M(icrosoft) A(mazon) G(oogle) cloud, yes it costs more.... But your time is worth it!

Why not run it in containers? What's the downsides?

I mean, if you can reliably run Kafka or Postgres in containers, what's sufficiently different about MongoDB?

Or are you talking about using Docker itself as the container host as opposed to K8s or ECS etc.?

I don’t get why the original response is downvoted. Of course you can run those in docker containers, but it is generally suggested that you don’t. From my experience i would run stateless services in containers and persistent storages in vms, dedicated servers, or cloud services.

People have been running database instances just fine on their own for decades, without the help of "big brother".

There are reasons why running a high performance database instance in containers is problematic, but security is not one of them - not any more so than application containers.

You just need to know what you are doing, it's not a black art.

Truth - I personally avoid doing so, but that’s only because I prefer not to introduce that extra layer of complexity into my stack due to lack of full understanding of the technology. There’s nothing stopping me from bothering to learn all the trade offs and pitfalls and doing so, but so far there hasn’t been enough of a compelling reason for me to go through that effort. I just go with the general advice “you usually don’t want to dockerize stateful applications” and leave it at that.

I was not referring to security, otherwise completely agree.

> it is generally suggested that you don’t.

What's the difference between a container and a instance in a cloud service?