- We're working on external store support. First implementation will probably be w/ Vault, but we would love for this to come from the community.

- If you have access to the managers, you have access to all the secrets (and access to administer the whole cluster), but normal swarm nodes only have access to the secrets required to run services that will be scheduled on them. Also this are erased as soon as the service is deleted or rescheduled on some other node.

That is the same trust boundary as in Kubernetes currently: https://kubernetes.io/docs/user-guide/secrets/

It is the most important step that you can package containers without having to know the production secrets and to have a "standard" was to retrieve them.

No one, without production access, has a way to obtain them.

Note exactly the same (for now):

> Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet

> If multiple replicas of etcd are run, then the secrets will be shared between them. By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.

Agree, it would be better if the nodes could not trivially request other secrets. But you can request new containers placed on any node and a root escape that works on one node almost certainly works on the rest.

We usually recommend subdividing the node acls by namespace when running disjoint node sets (where tenant A can't schedule onto tenant B's nodes). More fiddly than it has to be in Kubernetes today.

We are currently facing the same problem and came to the same conclusion, we are subdividing our node pools by namespace (and using namespacing for multi-tenancy) but it "feels" very convoluted and a smell that there's a feature missing in Kubernetes itself to handle this more graciously.

The node placement admission controller was definitely intended for that. In combination with a simple controller for allocating namespaces to sets of nodes.

The real solution is of course to limit what nodes can see - I won't say it's trivial, but it's an O(1) check based on the pods scheduled on that node.

I think in this case you'd already be compromised if you have to worry about a user being able to essentially maliciously place a container. Really a job for RBAC and not the scheduler itself.

I'm referring mostly to users who are legitimately able to run workloads on a non-zero percentage of the cluster. It's fairly hard to defend against a user with the ability to schedule even a limited amount of workloads, because good scheduling usually spreads (in which case you can compromise more machines and then take them out of service briefly) or packs (in which case you get put on nice target rich environments).

That's partially why we (openshift) just don't allow containers to run as root by default - it sucks for new users (most images assume root) but it dramatically lowers the risk of compromise across the board in any scenario with multi-tenancy. Agree that with isolated nodes per tenant you wouldn't have this issue, which is a more straightforward RBAC story.