Sounds like these OPS teams don't know what they are doing, but I'm not sure what you are saying is correct. Conceivably, in an ideal situation, the org has an internal intermediate CA created using a cert signed by a real root CA, and that intemediate CA's cert is in every client machine the org provisions to the employees, so the employees' machine can verify the server they are connecting to is authentic. For mutual authentication, the client certs lets the servers authenticate the other way around. Using a straight up self-signed server cert is a one way mechanism from server to client, it's easily spoofed, this makes no sense to me.

its nonetheless the way its being done. a few examples:

* remote docker daemon execution

* setting up a kubernetes cluster with kubeadm and creating certificates for remote kubectl execution (this one might be related to the previous one. I'm not sure as i don't know anything about the kubernetes internals.)

* rabbitmq cluster iirc. At least thats how you're supposed to set it up with the sensu monitoring framework

* previously mentioned puppet does its as well, though the process is mostly invisible to the user

i'm not sure how its supposedly possible to spoof though?

the master needs to sign a certificate for clients. IF the master is compromised, everything is compromised. Thats true no matter which authorization protocol you're using

I don't know how in your experience how these orchestration servers are setup, but in my experience, these clusters are all on the Cloud, which means accessing them requires going out to the public internet. In the off chance someone is actively MITMing the client's connection to the server, which isn't exactly hard, all that the attacker has to do is to present any cert signed with the same subject and issuer and whatnot except the public key. How would the client know if all the restriction is to accept a self-signed cert? There's got to be some sort of pinning mechanism like OCSP must-staple or verification step the client has to do to authenticate the server or the entire org's kubernetes client will be DoS'ed. I don't even have to compromise anything. You are saying self-signed server cert is how it is typically done, I'm questioning whether that's how it's supposed to be done.

The kubernete's document says:

  For an identity provider to work with Kubernetes it must:

  1. Support OpenID connect discovery; not all do.
  2. Run in TLS with non-obsolete ciphers
  3. Have a CA signed certificate (even if the CA is not a commercial CA or is self signed)

I interpret it as saying you should perferrably use a real CA or a well-behaved internal intermediate CA. Anybody remotely familiar with how TLS works will probably tell you the same. If what you are saying is correct, it's a serious problem in our industry.

Well yeah,it's essentially pinned... The client needs a cert signed by the master after all. If you exchange the master cert, the signed certificate is no longer valid

You only need a 'real' CA if your service is public facing.

Heck,even a Windows AD server uses self signed certs

/Edit: after re-reading your comments I get the feeling that were really talking about different things. The self signed certificate becomes the base for your chain of trust. Everything signed by this is allowed to talk to the master. This verification happens on both sides. A client getting an unexpected certificate will refuse to talk to the master just as the master won't talk with a node, unless it's got a signed client cert.

Because the master cert is self signed, it's easier to just throw everything away and recreate the hole chain of trust instead of bothering with revocations.

Finally, there is no benefit in using an official CA if you are in control of everything accessing the resources.

Am I making sense now?

This makes more sense, but I'm still confused by that self-signed bit. By master cert, if you mean that's your internal CA's cert that's distributed across every client and server as a part of your trust root, than this makes sense. If you are saying the master self-signed cert is served directly by the orchestration cluster, than this doesn't make sense, as the clients have no way of distinguishing one self-signed cert from another.

there is no point in importing it. The system store is never referenced in this setup.

The master (this is the self signed part) cert becomes part of the client certs by signing.

These client certs are now only valid with the original self signed cert.

It's essentially a CA, only for the client certificates. It just isn't formalized because it's never imported. you'd also lose the ability to rotate the certs quickly without gaining any security by importing this self signed certificate as a CA.