Probably would have helped a bit if their java server process wasn't running as root as well.

No one sane runs confluence as root. But RCE is RCE. It's always serious, even if the user isn't privileged. It's significantly easier to get root access once you have access to an unprivileged acccount than without. And with tools like Confluence, there's a good chance that if you control the application, you can intercept login requests and steal credentials. Which can then likely help you gain access to other services.

It would help if the run-user was not the same user that owns the files on disk. The run-user should only need to write logs and caches and databases. Not jsp files.

I usually try and make this so, but it is often very hard to impossible. Vendors do not make it easy.

>No one sane runs confluence as root.

See "Figure 1. Process tree in Volexity Volcano Server from the infected Confluence server"

https://www.volexity.com/wp-content/uploads/2022/06/Figure1....

I didn't mean running as non-root would fix the problem, just that a remote root RCE was worse than a normal RCE.

> And with tools like Confluence, there's a good chance that if you control the application, you can intercept login requests and steal credentials.

I thought Atlassian products were well trusted. Is that not the case?

They've had a string of script-injection vulnerabilities with similar severity -- the last was less than a year ago:

https://confluence.atlassian.com/doc/confluence-security-adv...

Each time, they patch the particular vulnerable script, but they don't seem to have done anything systematic which would eliminate the entire class of vulnerabilities (like, say, adding taint-checking for unquoted, client-supplied content before running code as an OGNL script). So they keep coming up.

They’re widely used, which is not the same. If you’re a customer you’ll routinely encounter bugs, clumsy or inaccessible UI, annoying limitations, and performance issues which take ages to get fixed if ever. I would not assume that they’re better at security than everything else.

Nailed it.

Let's also admit that setting up iptables is harder than it needs to be. I mean, not after the first twenty times... you eventually remember all the arcane logic and expect everyone else to as well.

If you're manually editing iptables rules in production in current year you're doing it wrong anyway, you should have some form of config management like ansible/salt/puppet etc that make it dead simple to add rules or at the very least just use firewalld or something

In the past, I've found making tomcat listen on unprivileged 8080 and proxying it using nginx on 443 works well.

Even better running the system as a Docker, or other, container image with read only filesystem.

who in the hell does this?

Mostly Linux-illiterate people, who want to bind to privileged ports (443), and have no idea about systemd sockets[1] and socket-activation[2], so they stick with the first solution that works.

Most people don't know that you can have systemd bind to the privileged ports for you, and hand over the fd to the port to your (unprivileged) process.

Same goes for docker/containerd btw, if you run docker as root nowadays, you are doing it wrong.

[1] https://www.freedesktop.org/software/systemd/man/systemd.soc... [2] https://www.freedesktop.org/software/systemd/man/systemd-soc...

False sense of security or just plugging one hole in a wedge of swiss cheese. It isn't always the server, sometimes it is just the framework. A proxy won't stop that.

https://www.zdnet.com/article/equifax-confirms-apache-struts...

The metaphor of plugging holes in a wedge of swiss cheese is actually the basis for the aviation industry's security model!

https://en.wikipedia.org/wiki/Swiss_cheese_model

It's more a risk management model than a security model. It tries to build a safe system from layers that are not 100% safe (e.g. because components on a plane may fail or people may not follow procedures exactly). That's why in an airplane accident it's not usually one single cause, but a whole string of things that have gone wrong. But still, we should try to keep the holes in the individual layers of cheese as small and as few as possible...

This is really neat and novel to me, thanks for sharing.

The IT version of it is sometimes called "defense in depth" - even if each individual layer has weaknesses, combining them gets something stronger.

This often doesn't apply in cryptography; where adding two separate functions may result in a system that is actually weaker to attack.

Well yeah, if you put two slices of Swiss cheese side by side instead of on top of each other, you have given the attacker a wider choice of holes...

Your link points to a vulnerability that could have been mitigated by putting the internal application developped with Apache Struts behind a proxy server like Nginx or GCP Identity Aware Proxy (IAP) that authenticate the request before the request is forwarded to the application server.

This only applies to apps for internal use like Confluence or Jenkins, not publically accessible websites.

Proxy preauth could be able to mitigate a vulnerable framework. The idea is that you never connect to the application server itself, but to a proxy that requires a separate authentication. That’s obviously not possible for a public-facing application.

Now, if the attacker has access to the proxy authentication, all of this doesn’t help much, but at least you could track and limit who can access what chunks.

My confluence server is behind a proxy which requires a valid client certificate to talk to the server

I have other proxies which require OIDC authentication before passing any packets on

Now sure, I'm still vulnerable to internal attackers from my company, but that's an audited trail to a real specific user, not just a random botnet on the internet.

Smaller attack surface internally, for sure. I also assume internal networks are hostile, not just from internal threats, but from external lateral movement.

I'm don't follow. If you have a pre-auth reverse proxy in front of your web application, you drastically reduce your attack vector.

An unauthenticated attacker can no longer simply rely on a vulnerability in the application or underlying framework, but must first obtain an employee's credentials, or find another vulnerability in the proxy or authentication infrastructure.

This isn't a false sense of security. It is a concrete additional layer of security which has tangible benefits.

I don't * - Too late to edit.

Plug enough holes though and at some point it starts to have effect.

The real problem is if there only one wedge, you need to combine that strategy with multiple layers of security to get anywhere, otherwise the first hole is the last.

Exactly, I mean everytime I saw any of this self-hosted SW, open directly to internet, my eye is starting to tick.

Agreed, if not a proxy then put your services behind a VPN or a bastion host when possible.

And the likes of Cloudflare access or VPN only access via Tailscale make it so easy to keep it protected. I'm always a bit shocked when I come across publicl ogins for Jira or Confluence instances of large corps.

Yup. Really easy to set this up with the IDP's available today too.

In the IdP/SP/SAML sense of it, I'm really used to that level of validation actually hitting the application and being processed through an authentication subsystem, rather than being done at the webserver level.

Possibly just some nginx features I'm not familiar with, or are there whole apps out there that serve to be nothing other than an authentication layer inside of a webserver? Are they popular?

Sorry if I misunderstood the question, sounds like you are asking if there are some common tools that do auth outside of the application before passing traffic back.

Google IAP is a big one, and Cloudflare Access can let you do similar things (link an internal service to Cloudflare network and Cloudflare will deny someone ability to talk to that service until they've validated their identity to cloudflare and it matches your rules somehow.

Azure AD Application Proxy fits the bill for Microsoft 365 shops.

Nothing from Atlassian should ever be exposed to the public internet. Atlassian stuff is a security tire-fire with very old dependencies even in the latest releases, a painful reinstall/config merge needed for even minor patches, and no auto-update mechanism.

You absolutely do, but I was more interested in the mechanics of it at the webserver level rather than SaaS equivalents. The Google IAP one is helpful as I only knew about Cloudflare Access.

HTTP Redirecting out to an idp and it posting back to the application with a SAML assertion is pretty common, if that's what you mean. Lots of people do that with things like O365, the AWS Web Console, etc.

Of course. Just asking how is the web router aware of whether or not the redirect is necessary if you're not hitting the application to validate login state.