You could also give an unprivileged user access to run VMs, that has its own set of security tradeoffs, but VM technology is generally considered more isolated and mature.

Unprivileged user namespaces give an unprivileged user the opportunity to use syscalls like chroot, mount, etc. which they historically couldn't use. Container managers like Docker and Podman drop those "capabilities" after creating the container, so an application running inside a container isn't that unsafe (probably, at least everybody is doing it nowadays for production servers). I think a more problematic scenario is that someone who gets hold of an unprivileged user can then create an user namespace without dropping those capabilities and then use a local root exploit based on those syscalls you normally weren't able to use.

I think the problem might be in the interplay of how many allications/daemons already try to drop privileges by running as a user after doing the few things they need those privileges for (like binding to a low port), and if you don't even need those privileges, you don't even need to start it as root, and your container has well known security properties.

If you're running rootless and using namespaces to allow non-root users access to previously root-only kernel APIs, then a bunch of prior assumptions may no longer hold, and there's a new attack space available to target that has always existed, but was of no use previously to exploit.