I have been curious about this, too. I think I'd want something like vault to issue OTPs that can be exchanged for secrets over a socket to a sidecar, where the OTP is made available as an environment variable set by the orchestrator (eg k8s). If the token is used twice, lock it all down. If it's read but not confirmed to have been received by the service (through some method... dunno), lock it down.

Thanks for coming to my brain dump.