I have a question about redundancy or "What happens if your gatekeper EC2 instance goes down"? If you have multiple gatekeepers could they be set up this way:
- let's say you have five different web apps using a gatekeeper to hold their secrets
- let's say you have n gatekeepers (let's say 3) and each of the apps knows the address of all three gatekeepers.
- If the primary gatekeeper is unreachable, all five apps would try to contact the secondary gatekeeper, but that gatekeeper would only (ever) respond in the event that the secondary gatekeeper also found the primary gatekeeper unreachable.
It's like a sleeper cell - at any given moment you have multiple replacement gatekeepers ready and waiting to serve, but each of them is unable to respond unless the one above it in the list stops responding. In this way you could lose gatekeepers (even permanently) and build a little bit of resilience into the apps depending on it while you're able to sort out what happened and restore normal behaviour.
Is this a good idea?
1. Create a JSON file containing encrypted secrets (DB pass, etc.)
2. Upload the file to a secure S3 bucket with fine-tuned permissions and server-side encryption
3. For the instance that is launching, include the permission in the IAM role that allows it to "S3:GetObject" on the specific JSON file you uploaded.
4. Deliver decryption keys to the app in some manner (Chef, Ansible, etc.)
5. When the app starts, it downloads the JSON file and loads the environment variables.
I wrote a blog post[1] about this with more detailed info and an NPM module if you use Node.js.
[1] http://blog.matthewdfuller.com/2015/01/using-iam-roles-and-s...