Similarly, the docker security model is that there isn't a security model. If you can talk to the docker socket you have what ever privileges the daemon is running as.
Second point, yep if you run Docker as root and someone can access the socket file they get root.
If that's a concern, you can run Docker rootless.
And as we're talking file permissions on a local host to allow that access, the same applies to podman containers does it not? If there are permission issues allowing another user to use access the container filesystems, you have the same problem.
As to swarm, I was comparing Docker rootless to podman, which is more a developer use case than prod. container clusters.
[0] I suppose you could have a user-level daemon that runs for each user that needs to run containers, but that's even more overhead.
There's some tradeoff I guess though, between rootful setup and per user, as images duplication per user could add up.