At the top level you have an organization account, which is where billing occurs.
From this org account you create accounts for the following (typically):
1. Security - AKA the account your USERS are in 2. Ops - The account your monitoring, etc are in
From here where a lot of people seem to deviate (I've been interviewing level 2-3 SREs for the last 3 weeks and have heard all about different AWS structures that I don't like) is how to break up your applications into their own accounts for a low blast radius.
What I DO, and is well known as being the best practice, is to create an AWS account for each environment of each application.
App1-sandbox App1-staging App1-production
Then your terraform is also structure by application/environment/service. Each environment and application has it's own state in s3 and dynamodb.
And so on.
Is this unwieldly? I have 40-50 AWS accounts and no it's not unwieldly at all IMO. Cross account IAM and trust relationships are set up very early on and they don't need to be modified much if any at all until you create another AWS account. Creating a new AWS account is kind of annoying, though. I need to automate that process better.
https://aws.amazon.com/organizations/getting-started/best-pr...
I haven't worked on GCP since maybe 2016-17 so I'm not sure how it's going over there anymore.
GCP native API is basically the same thing as knative in most ways. Just a bunch of various services and resources that you all call and authenticate and even often provision the same way. It’s very predictable, it’s very easy to model things on top of and to feel fairly confident that I’m not accidentally doing something stupid so I really like it from that point of view.
You can do that with terraform...
Right now we’re using one account per env but also see downsides and thought of going the next step to do one account per env and tribe/team.
It's not for the faint of heart though. You need to allocate subnets to individual applications (with relevant capacity planning concerns) plus support is sometimes spotty (e.g. EKS doesn't support it last time I checked). Not worth doing until you have several teams trying to use the same VPC and stepping on each others' toes.
For our EKS apps we do helm rollouts, but most of our services are on ECS so it's mostly just updating a task definition and forcing a redeployment of containers.
Each EKS cluster is set up exactly the same aside from the usual things like vpc and ips and things of that nature that switch between them. They all get a set of "base" apps like log chutes and cert manager and all that as soon as they're deployed.
Our app environments don't communicate with one another at all. The only relationship between them is our IAM accounts in our security account can assume access into them as admin/etc.
There are AWS systems above the account level for managing it (Organizations), so its not quite as bad as it might naively seem, but, yes, its more unwieldy than GCP’s projects.
Billing works by having a billing aws account that all other accounts are in a sense "children" of.
AWS IAM doesn’t do capability inheritance; if I can write a policy at all I could grant any privilege to any resource in that policy, even privileges I don’t personally have. It’s easier for each groups of admins to have a separate AWS account than to put everyone in security boundaries that try to wall them off.
The same thing happens with the Kubernetes provider when you try to use it with multiple GKE clusters.
But I’m coming at this from a GCP lens and got half way through the article about how the recommended unit of isolation in the AWS environment is entirely different AWS accounts and I’m kind of hung up on that. Is that really a thing people tend to do often? Doesn’t it get super unwieldy? How does billing work? What about identity? I have so many questions.
EDIT: Despite the fact that the root resource in both GCP and AWS is an organization, when I heard “account” I mistook that to be AWS terminology for an organization.