(I should also point out that this is in a batch ETL system.)
The nice thing about this approach is that it is virtually impossible for secrets to end up in source control. The risk is that the vault becomes a major single point of failure.
I think you don’t even need to redeploy any long-running tasks. If they have a connection open, it should continue working, and if they don’t, they should consult the vault when they open one and get the new credentials from it (this may be different for different kinds of credentials)
There will be a race condition between creating credentials and storing them in the vault, but even that can be avoided:
- create new credentials with same rights as the old ones
- update vault
- delete old credentials
Example, if it's AWS it may rely on ec2 instance role, to allow it access to the secret manager.
If it's kubernetes, it can be done through k8s token mount, basically allow token in namespace access the vault, and the token(which is generated and manage by k8s, which is just a JWT btw) is mounted into your pod.
I'm not the person to ask how to set it up but it's seamless in use and can return data in json format which makes it easy to parse. If it were my money I'd look very hard at this product.
Even unsetting environment variables leaves them in /proc/self/environ. There isn't a thread-safe way to unset environment variables in POSIX, so even if you `unset TOP_SECRET` from within a program the contents of /proc/self/environ will remain unchanged and available.
> Once the application hits “readiness” status (as determined by health endpoints or the load balancer), the secrets volume should be unmounted and made inaccessible.
> Ultimately, secrets need to live somewhere and need to be accessed as plain text. Just make sure that the access as small window is as [sic] possible, and try to obliterate it after use, if possible.
This is not an all-or-nothing situation; it's a game of mitigation. If the process needs to retain the secret in memory, true, there's not much you can do about that. But I don't think that minimizing where else you're storing it is just security through obscurity. Fewer possible attack vectors is still fewer possible attack vectors.
The last time I was responsible for the scaffolding of a new software service it supported reading secrets from a file. It confused the operations team and I was just told to include support for an environment variable. Oh well, I tried.
> If your app can’t interface directly w/k8s, but it can read secrets from a file, you can use a small init program to fetch the k8s secret and write it to a named pipe. This is advantageous compared to mounting as a volume, because the pipe disappears after both ends close their connection to it.
Reminds of the "Front fell off" comic sketch. "No, we towed it out of the environment"
"Nothing's out there."
"Well there must be something out there."
"There is nothing out there. All there is, is sea, and birds, and fish."
"And?"
"And 20,000 tons of crude oil."
"And what else"?
"And a fire."
"And anything else?"
"And the part of the ship that the front fell off. But there's nothing else out there. It's a complete void."
The 12F section on config already mentions quite prominently the use of config files.
Taken from https://12factor.net/config
> "Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific."
now we are just moving from trusting a bunch of distinct services to trusting this single broker... just moving the responsibility of trust to a single point of potential failure no?
Also dont credentials have to be passed to secretless broker? how does it know the application has access to the service? isnt that still at risk of being leaked.
i like the idea of not thinking about secrets but it seems to good to be true.
Not having long lived secrets is the ultimate destination, but we all live with the legacy around us.
The pros/cons of environment variables vs. files (or other approaches) is also something I’ve thought about a lot while working on EnvKey.
We use environment variables as a default approach, since it seems to be the most common way to pass secrets/config to a process in the wild and we want to meet people where they’re at. But we also make it easy to use files or system calls instead (I think system calls are actually the most secure.)
One thing it’s always important to remember though in security: if you make the “secure way” too hard, people will route around it, making matters worse in practice. There’s always a balance to be struck.
Honestly, I’m skeptical that threat models where environments are exposed but files are safe are realistic enough to be worth worrying about. At that point, it seems like rearranging deck chairs on the Titanic.
It seems simpler to say “our last line of defense is the OS boundary.” You trust the host and go from there. If the host is breached, you’re screwed in a plethora of ways. There’s no point in sweating the particulars. Just don’t let it happen in the first place!
And when it comes to concerns about leaking the environment to sub-processes, this seems like a deeper problem. Even if you don’t store secrets in the environment, that doesn’t mean it’s safe to just send env vars off wherever. At this point, like it or not, environments are sensitive, because enough people and programs treat them as sensitive that it’s a self-fulfilling prophecy. If they might be leaked, then the leak is the security problem in my eyes, not the data in the environment.
Only one thing strikes me as unfortunate, which is it seems like it fragments the development process because now your code history is tracked in git but your config history is in EnvKey. It means everything I do is duplicated (for example, tags, branches etc). It isn't clear to me how I would take, for example common operations such as rebasing one git branch on another and replicate the same on the two EnvKey branches involved.
We track all our non-sensitive config in git for this reason - then merging two branches is literally merging the config changes. However, we don't have any of the nice features EnvKey is doing like service restarting, inheritance, secret management etc.
Just curious if I'm missing something here or you have a strategy for what I discuss above?
I definitely see your point on splitting up the history!
One thing to consider though is that you're already forced to split up the history in order to handle secrets securely.
Given this unfortunate truth, I think you're better off storing all the config and secrets together rather than splitting them up and keeping config with the code.
There's not always a clear line on what's a secret and what's "safe" config, so attempting to split them up like this is asking for trouble: you're almost guaranteed to end up with secrets in your git repo eventually.
All that said, we've thought about adding a git integration that would output a non-sensitive state file ala 'tfstate' so that you can see your EnvKey history baked right into your git history--it wouldn't show values but you could see what keys had changed. Would something along those lines help to address your concern?
You're right about the split situation for secrets. I guess secrets have a different lifecycle generally anyway (just cos you roll back your code doesn't mean the password to production should be what it was at the time of the prior release). For this reason they (nearly) always sit outside of the versioning process for the code.
However our secrets are probably less than 1/10th of the config and even less of the complexity. So I am content to use a separate secrets manager / process for that.
Would be great to think about how to integrate with git workflows. Seems a bit messy but something like you suggest where it dumps it out to a file could be a way. Would be good to solve as incorrect / missing config is one of our major reasons for deployment issues, and so far despite all its evils, tracking them in source control as part of the dev process has been the best solution to minimise it.
If your app can’t interface directly w/k8s, but it can read secrets from a file, you can use a small init program to fetch the k8s secret and write it to a named pipe. This is advantageous compared to mounting as a volume, because the pipe disappears after both ends close their connection to it.
Is there a standardized/best practice way to do this? Some convention for file names or format?
If you use pydantic, it supports a secrets dir with a predefined path.
https://specifications.freedesktop.org/desktop-entry-spec/de...
A secrets manager may be a better option, these days. I don't believe they were a thing in 2011 though.
Also 2011 was 11 years ago, and I still keep this webpage in mind when building server-side software <_<.
I’ve been trending towards treating secrets and non-secrets differently.
For non-secrets, I tend to have a config discovery mechanism that lets me read config from files, then from env, then from cli args as final fallback. This is mostly for ergonomics. I prefer to use files for deployed processes, and env vars or cli params when debugging or working interactively. Env vars are handy for cli args I’m going to be repeating a lot.
But for secrets I’ve tended to only fetching those from files, but allowing the file location to be specified by env var or cli arg for quick changes.
It sounds kinda complicated, but you write that logic into a library and then everything behaves the way you expect consistently.
We use these for running locally and debugging as you say, but come deployment time we ship all applications as helm charts with default config values, store overrides and static secrets (with SOPS) in a different, company-global repo, and use kubernetes secrets to mount a tmpfs volume to pass them in to the application. There's no way to pass args or env there.
I’ve generally found it much better to mount these into the filesystem and read from there. It helps with, for example, secret rotation for long running processes. Relying on process restarts can be ugly in some setups, especially if startup time is expensive.