Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs.

As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ...

I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.

All internal services should be protected, even on your home network.

If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.

At every layer of “breakage”, you are cutting down the chance for an attacker to access your system. While it’s possible the failures might cascade it seems likely that these failure modes are pretty independent of each other.

You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.

I think you misunderstood me - I mean that you want multiple layers to your security, and you need to be careful to select layers that fail independently. If Vault has a sufficiently bad unauthenticated bug, then attackers can simply use it to request database credentials and query the DB, which is now on the open internet.

An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."

My view is that for most cases the cost/benefit of multiple layers doesn't stack up. Given a fixed amount of available time and effort, you'll generally get better results by focusing that effort on making one really good layer - e.g. putting active monitoring in place so that you detect when your single layer breaks (whether that's an attacker from outside the network being able to connect to inside the network or an attacker with an expired certificate being able to connect to a live system) - than by spending half as much time each on two layers, IME.

Unless this is literally running on a single 1U host in some colo, there is no excuse for not having defense in depth. An old school DMZ if you are in a datacenter. A VPC if you are on the cloud. Then client certs for everything. Two factor for ssh. Auditing. These are straightforward to set up, with different options if you have time-but-not-money or money-but-not-time. If you have not-money-and-not-time then this is probably going to end badly no matter what.

> there is no excuse for not having defense in depth. An old school DMZ if you are in a datacenter. A VPC if you are on the cloud. Then client certs for everything. Two factor for ssh. Auditing.

Maybe there is no excuse, but literally every company I've worked for (including Fortune 500s) has been missing at least one item from your list. So "industry best practice" means committing less time and money than it would take to implement all those things properly (rightly or wrongly) and you need to triage and prioritise.

So the excuse is "my company doesn't take infosec seriously". Like every similar issue of "should do but don't" (testing, formal promotion processes, diversity, harassment response), you get to decide to tolerate it or get a job elsewhere. My experience is that the companies that take this stuff seriously also do a better job of converting my skills into cash, and as a result, pay me better.

Interesting. My experience is that companies that "move fast and break things" have been better at making money and paying me (whereas I personally lean too far in the perfectionist direction). Interesting businesses face a wide range of risks - competition, regulatory, market - and infosec is rarely the biggest one IME.

I just want to say that I think this is absolutely a fair approach for many systems, but it sounds so radical against the backdrop of perimeter security that you're catching undue flak. You need to take a lot of care to build and design the "one layer to rule them all" in a way that's ridiculously sound, but this can actually produce a strong design if you pull it off. I didn't give this advice because I looked at where NewsBlur is today and figured they had some learning curves to get over before they could make the right tradeoffs to do it safely. If they got that much wrong with their DB, they might not know how to design a sane zero-trust network, or even how to make rational decisions in that world.

You see this "authN/authZ above all else" line of thinking in Google's security design [0], with their ubiquitous login wall. For their employees, that login wall has extra hardening - you can't do regular password resets, and you need to possess a physical security key (which acts basically as a scaled-down single-purpose HSM) and pass a suite of posture checks on the device (proportional to the sensitivity of the protected resource) to pass through it.

Then they put this login wall in front of everything, even internal services, and that tends to be OK because the system "fails closed," and the only way to access their protected resources is via physical safe rooms deep in the Google offices in which elevated privileges may be obtained.

Putting all your systems on the Internet forces you to get the incentive structure right, and that can be useful in companies where "private networks" can serve as justification to weaken security - if your login page is Internet-exposed, then you simply have no choice but to make it strong enough to withstand the chaos of the Internet. There's significant merit in that.

[0]: https://sre.google/books/building-secure-reliable-systems/

> 2. Use real authentication rather than network firewalling.

This is in vogue but it’s wrong because it presents this as an either/or.

In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone).

There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges.

People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.

In reality time and attention are limited resources. Does network-level filtering have a good RoI relative to putting the same effort into improving security at the service level? IMO no (at least not until you've reached a very high level of service security where you're hitting severely diminishing returns) - network-level security is necessarily at least one of a) crude b) complex enough that it becomes an attack surface itself - though reasonable people can disagree.

You are looking it as a network level filtering.

I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it.

In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public.

This is easily, done on something like AWS.

The very concept of a "private network" leads you down the path of making a security boundary that has far too large a surface area, IME. Either it becomes a big bag with all your hosts in it, and you write systems that trust all the requests they get (even if you know you "shouldn't") because you know only your systems are on the network, and then an attacker figures out how to make one of your systems make an arbitrary request and you get owned. Or you put each component on its own private network but then you have to open up every port you're actually listening on so that your components can talk to each other (and you probably automate that in your kubernetes/puppet/whatever setup, like what happened here with docker) and the private network does nothing.

An independent firewall accomplishes the same even if every server has a public IP.

Respectfully, all of the suggestions are good, as are your two. But, segregating your public traffic and private traffic is an excellent way to prevent these sorts of issues from happening. While it doesn't prevent someone who's compromised your network from accessing the DB, it means someone can't fat finger a port and open a DB to the public internet.

Well, your second point is his first point.

And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.