If you want secure config storage on Kubernetes, you end up using Secrets, which ends up being key-value like env vars anyway. If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.
You can load the secret file directly into the app, no need to load it as env vars or keep it strictly as key-value pairs.
> If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.
You can use tools like ejson[1] or sops[2] to get encrypted files checked into Git that have key level granularity on diffs.
There is definitely a place for higher level abstractions than Kubernetes. Mostly it gives operators a standard platform to build from when teams outgrow the PaaS sandbox.
[1] https://github.com/Shopify/ejson [2] https://github.com/getsops/sops
Now you're getting away from the spirit of 12factor and hard-coupling them again. The intent is for the app to consume the secrets but have no knowledge or care where they came from.
Edit: misread as "load the secrets directly into the app." Yeah, this is just env vars but different.
1. create a Secret that defines the file as a key
2. mount that Secret into the container (.secret.secretName in the volume definition) under an arbitrary conventional path, e.g. /secrets
3. define an env-var that gives the path to the file; where the assumption when this env-var isn't defined is to find the file in the working directory. If your own code uses the file, it can read the path from this env-var; while if the core of your container is some opaque tool that doesn't know about the env-var but does take the file's path on the command-line, then you can wrap the tool in an entrypoint script that reads the env-var and passes it as a command-line arg to the tool.
IMHO, this approach is flexible but principle-of-least-surprise obeying for both development and production deployment.
A .env file allows you to go from knowing/assuming where a particular file is, to having environment variables.
What I'm describing is that you can go from being passed environment variables directly, to knowing where certain files are; which therefore allows those files to be mounted into a container-image in arbitrary places, rather than encoding brittle assumptions about config-file locations into the binary within the container-image — assumptions that usually get in the way during development, when you're not running the binary inside a container.
The entire point here is that 12-factor isn't the be-all, end-all, so this is irrelevant.
> The intent is for the app to consume the secrets but have no knowledge or care where they came from.
This doesn't really make sense, and is an impossible requirement. The app will always have to know where configuration or secrets come from. Environment variables is just one option of "knowing where they come from". Choosing file storage is just as valid.
We use a custom setup in which the config values are loaded from multiple sources actually, so we could put everything into secrets manager, or load them all from env vars or HOCON files.
If we had to set every config value in ECS / lambda using env vars, it would be a major pain in the ass, and error prone.
And how your apps authenticate to Secrets Manager ? Did you ever call `env` on a pod that has IRSA configured ?
This is just a middle step to do exactly the same thing but instead of using envFromSecret you use envFromSM
That if you dont want to pull in k8s dependencies to the code, at the end every approach ends up as a mounted file or env vars.
The Twelve-Factor App is marketing content produced by the former Founder & CTO of Heroku. It's a great piece of content no doubt, but it should be viewed with that lens.
Not all marketing is that straightforward.
I'm fine reading a few passwords and tokens from a secure key-value store. But the rest of my config can be a bit more expressive.
Doesn't break diffs, doesn’t directly couple anything.
Configure your secrets in Azure Key Vault (AWS Secrets Manager, GCP-something-I-forget) or whatever the non-cloud Vault software is called.
Devs only set client_id, client_secret, and env defaulted to dev. On startup, the app fetches configs for the secret store. The needed secrets are in source control. If the needed secret is missing from the secret store, there is a clear error on startup.
You are welcome, I just saved you weeks of onboarding, confusion, and devs passing around .envs on Slack.
Don't get it. At one company I worked, .env files for the DEV environment were open (the file itself was stored on Vault, so only engineers had access to it), and one could only access DEV resources via VPN. So, starting a service using the DEV .env file was rather easy an uncomplicated.
My proposal:
1. Key Vault | HashiCorp Vault | whatever is the source of truth for secrets.
2. When a developer adds a key, they add it to Key Vault and update code to use it.
3. On app startup, code checks the dev's local for 2 env vars ONLY, which allow access to key vault.
4. The app then tries to pull in all the env vars it needs by secret name.
5. If any are missing, it fails.
This keeps devs from having to pass around .env files outside of source control. Btw, Azure (and all other cloud) builds can be configured to pull env vars directly from Key Vault, so there's that benefit too.
That's literally all you need to solve the scenario you described with .env files.
So with .env file:
- developer adds to .env
- pushes code
- all developers see failures in dev env due to missing env var
- all developers search slack/email to figure out what went wrong
- all developers update their .env file and things are working again
————- In gp’s comment ———-
- one person works on feature that introduces new config
- failure happens in their local environment
- they update config backend and push code
- all other developers get updates with no changes needed
I do and have done both and a blend. Secrets for external services needed in dev usually in parameter store. Kinda depends on the existing work flow. There is so much to do you should be tackling problems that move the needle and chip away at small stuff, pet peeve stuff in smaller time-boxed increments.
It depends, always it depends.
We use SOPS and the encrypted .env file is stored in VCS. The key is stored in 1password with instructions on where to stuff it on your machine. Then you just decrypt the .env file and you're off to the races.
My company open sourced the .NET library we use for this: https://github.com/Lokad/Secrets
If you remember the world before 12 factor you know that even if motivated by self serving incentives, Heroku making this mainstream has been an overall win for the entire industry and we are generally better and more secure for them having done so. We are still benefitting from all of this today, and Heroku who championed it is making death rattles.
Here is some obvious case observed many times.
A small and very fast pace team builds a server application having limited to no external dependencies. No db even.
Just a server doing stuff.
It does have some config yes, many many key/value pairs even, a config file ended up with a dozen properties just to control Req per sec limits and such.
Now just to deploy the thing, since of course env defined configurations is where things belong in best practice, the dev team setup a process to peer review those env values and despite all of that, fail to get the app running.
Human errors happen but since more mistakes and release delays were caused by misconfiguration than any other reasons, the keen and bright enginnering team all agreed it would be better to have a config repo. One genius in the group argued that having a config service would be even neater than have the app read from a git repo! The whole got so hyped and of course spend the whole next sprint building and testing that. Brilliant, it worked!
Except that of course outages happened, of course. Outages happen.
Bringing down all environments one by one, hence blocking pretty much everyone. Everything went down because hey, since we made that effort which took for longer than we thought to build a config service and have our app reads from it, one of the dev minds thought it was a genius idea to have the app reload the config every 5 mins at runtime so that if a configuration change was made, no need to even restart anything. Of course error handling wasn't robust and well tested since the boss reminded the team we've got to ship a product by the end of the quarter and solving the universe can maybe wait 5 or 10 years after the IPO.
Team resorted to simply track the config in the repo. Kept the runtime reload of the config and things worked just fine.
Things can be done right and impacts could have been avoided. The example may seem like a twisted case since these were all configurations, not environment specific values. But I've just seen so many occurences of env variables being separated from the application repo costing so much and distracting contributors for no benefit that I have grown very cautious of it.
If you store the database hostname/credentials, as an example, in the "code" (either literally in the app code, or in a config.ini that lives in the repo and gets baked into a release), then the ops people can't quickly repoint the app at a hot-spare database during an incident: they'll need to edit the repo, push it, cut a new release, etc. If there is a long build step (as for a C++ app, for example), then this isn't really tenable during an outage. If this is in an environment variable, their normal deployment scripts can make the change for them.
I agree that this is probably not necessary anymore if your team is all wearing both hats (dev and ops), and its just as quick to make an edit to the repo and cut/deploy a new release as it would have been for the Ops team to run their deployment script in the old days.
Even when ops are perfectly comfortable with git, they are rarely keen on touching dev repos even if deployment times were tenable. If code changes need peer reviewed, how so configuration values don't get treated at least the same way?
Secrets in clear are obviously a no go, I feel though that encrypted secrets in repos has become reasonably simple with mature tooling in this area. Ansible's fast growth got cut short once people started to hear k8s at every meet ups perhaps.
Can I ask what the modern solution for this is?
Also:
> If code changes need peer reviewed, how so configuration values don't get treated at least the same way?
The easy answer to this is a CODEOWNERS file, with the app code being owned by the devs, and the config being owned by ops. But again, not sure how to do this for secrets without their being plaintext in the repo. Do any code review tools support encrypted content?
I think that's fine. 12-factor is a set of guidelines about what you do in production. There's a whole universe of things that are smart ideas for development and bad ideas for production. Leaving debug symbols in your binaries, leaving ports for connecting a debugger open on your containers, etc.
You just set the variables at startup in prod, and in the file on development (probably mostly because you want to change them all the time).
There's an enormous difference between "I'm going to use a dev-friendly way to manage and swap out all the different configurations that I might want to plug into the app while I'm testing it, but still plug them in using the same mechanism", which is basically what you get with dotenv files, and, "I'm going to configure with environment variables in production and JSON files in development," which is another thing I often see people doing.
... your secrets get checked into git.
This way, the same source deployed in, i.e. your AKS cluster in Azure EU north will take the configuration you set for that cluster while the one you run out of an Docker Swarm of RPi Zero's in an Ikea photo frame (I have one cluster like that, called Gibson), will take the configuration from that cluster.
> The twelve-factor app stores config in environment variables (often shortened to env vars or env).
Why are they fine? This seems to be an argument grounded in "it's a newer norm therefore it's better" but I don't see the justification. I've never used Heroku but ConfigMaps are a bane to manage. Even as a general programming principle within application code, colocality is a well-known, long-discussed topic around improving maintainability. If you're not using monorepos, that's currently not a reasonable option with current orchestration conventions.
Even if you are using monorepos, you're highly likely to either get into writing & automating DSLs to feed into your configmaps or do weird things like using Jinja to do file includes in your mounts. That's all before you run into the etcd 1MB size limit & realise that actually this whole system is a hack.
TL;DR newer ain't intrinsically better
Then in the case of a kubernetes setup. The application is wrapped inside a mechanism that gets the configuration from some where else.
- Logs as a stream: yep, just write your logs to STDOUT instead of a file. Let the orchestrator deal with reading and storing it.
- Configuration in the environment. Apps tend to pull config from different sources depending on the deploy - maybe a .env locally and a secret store in production.
- Port binding - your app listens on a port and you put nginx or whatever in front of it and set up a reverse proxy. K8S services and ingress do that.
I think the biggest criticism you can lay down on 12 Factor these days is that it's not actually written very well and assumes you know exactly what it is talking about.
Apache for my personal projects, in front of the docker containers.
This is one that always feels odd to me: what happened to rsyslogd?
The normal output of a non-service command (eg. grep) sometimes/often goes to STDOUT, which is why that command will write errors to STDERR -- even "info" messages that look like logging.
> Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific.
> The twelve-factor app stores config in environment variables (often shortened to env vars or env).
yeah the reason they argue for this is that the people that wrote this worked on Heroku, and the way that Heroku worked is you populated the environment variables from some fields in a web app. If you want your config history tracked in version control, or you do gitops, or you have k8s configmaps, or you want to have your configuration files on a mounted volume ... those things are all broadly fine; they keep the configuration state separate from the app deploy state. This document really confuses the forest with the trees and recommends things based less on actual engineering principles and more on the product capabilities of the corporation that produced it. It is an actively harmful set of guidelines.