News for hackers. Stuff that matters.
Re:Terraform best practices for reliability at any scale by mdhb on Friday August 04, 2023 @04:55PM (#37005760)
Sorry if this comes across weird or snotty it’s not supposed to.

But I’m coming at this from a GCP lens and got half way through the article about how the recommended unit of isolation in the AWS environment is entirely different AWS accounts and I’m kind of hung up on that. Is that really a thing people tend to do often? Doesn’t it get super unwieldy? How does billing work? What about identity? I have so many questions.

EDIT: Despite the fact that the root resource in both GCP and AWS is an organization, when I heard “account” I mistook that to be AWS terminology for an organization.

Re:Terraform best practices for reliability at any scale by swozey on Friday August 04, 2023 @05:02PM (#37005850)
The way this works with AWS is similar to you making a GCP project.

At the top level you have an organization account, which is where billing occurs.

From this org account you create accounts for the following (typically):

1. Security - AKA the account your USERS are in 2. Ops - The account your monitoring, etc are in

From here where a lot of people seem to deviate (I've been interviewing level 2-3 SREs for the last 3 weeks and have heard all about different AWS structures that I don't like) is how to break up your applications into their own accounts for a low blast radius.

What I DO, and is well known as being the best practice, is to create an AWS account for each environment of each application.

App1-sandbox App1-staging App1-production

Then your terraform is also structure by application/environment/service. Each environment and application has it's own state in s3 and dynamodb.

And so on.

Is this unwieldly? I have 40-50 AWS accounts and no it's not unwieldly at all IMO. Cross account IAM and trust relationships are set up very early on and they don't need to be modified much if any at all until you create another AWS account. Creating a new AWS account is kind of annoying, though. I need to automate that process better.

https://aws.amazon.com/organizations/getting-started/best-pr...

Re:Terraform best practices for reliability at any scale by mdhb on Friday August 04, 2023 @05:04PM (#37005872)
Cool, that was a genuinely fascinating window into AWS for me. Thank you for sharing
Re:Terraform best practices for reliability at any scale by swozey on Friday August 04, 2023 @05:07PM (#37005905)
FWIW I loathe AWS IAM and miss GCPs organization.
Re:Terraform best practices for reliability at any scale by mdhb on Friday August 04, 2023 @05:11PM (#37005942)
I’m quite into learning a lot of cloud native security stuff and I have to say my first impression was that it seemed so much harder to think about creating a secure environment using AWS IAM. I couldn’t tell if it was just a case of familiarity or not.
Re:Terraform best practices for reliability at any scale by swozey on Friday August 04, 2023 @05:25PM (#37006105)
I'm sure it's because of it's age and them kind of creating their version of IAM from scratch (someone correct me if they copied this structure from elsewhere) but you have to do a lot of goofy obtuse work with IAM automation. There are times I have to go into the console/cli and grab some sort of specific UID for an object instead of using its name, things like that that just make it annoying. Sometimes you can't use an account name and have to use the org ID... I could go on. You just kind of deal with it.

I haven't worked on GCP since maybe 2016-17 so I'm not sure how it's going over there anymore.

Re:Terraform best practices for reliability at any scale by mdhb on Friday August 04, 2023 @05:38PM (#37006232)
It really does sound like an entirely different level of complexity.

GCP native API is basically the same thing as knative in most ways. Just a bunch of various services and resources that you all call and authenticate and even often provision the same way. It’s very predictable, it’s very easy to model things on top of and to feel fairly confident that I’m not accidentally doing something stupid so I really like it from that point of view.

Re:Terraform best practices for reliability at any scale by wpietri on Friday August 04, 2023 @05:50PM (#37006340)
Yeah, I started with AWS and then spent a year on GCP and next time I'd much rather do GCP. It felt much more manageable and supportive to me.
Re:Terraform best practices for reliability at any scale by androidbishop on Saturday August 05, 2023 @02:09AM (#37009430)
A rare opinion but one I share wholeheartedly. I started my career at Google Cloud but spent the rest of it working with AWS. AWS always feels like an uphill struggle, lots of micro management and resources that need to be duct-taped together. I'm lucky to have recently landed a Google Cloud gig and my God, things are so much easier and smoother now. It just seems better designed and integrated to me, albeit much fewer services to choose from if you don't buy into their ecosystem.
Re:Terraform best practices for reliability at any scale by mdaniel on Friday August 04, 2023 @07:32PM (#37007228)
FWIW I loathe GCP IAM and miss AWS IAM, CloudFormation, and not having to talk to any one single person or piece of software about "please enable this foundational API in your Project"
Re:Terraform best practices for reliability at any scale by infogulch on Friday August 04, 2023 @09:43PM (#37008133)
> Creating a new AWS account is kind of annoying, though. I need to automate that process better.

You can do that with terraform...

Re:Terraform best practices for reliability at any scale by yellowapple on Friday August 04, 2023 @10:04PM (#37008261)
You can, but using Terraform to provision resources inside those accounts entails pulling generated/defined credentials from the org-level TF state and feeding that into the provider config for each app-env-level TF state. Vanilla Terraform doesn't support that very well (or at all, last I checked), but either some CI/CD pipeline creativity or Terragrunt (or both!) can work around it.
Re:Terraform best practices for reliability at any scale by infogulch on Saturday August 05, 2023 @04:02PM (#37015801)
terraform_remote_state
Re:Terraform best practices for reliability at any scale by yellowapple on Sunday August 06, 2023 @04:10AM (#37019928)
Can providers use the output from terraform_remote_state to set e.g. credentials? Last I checked, datasources get sourced during terraform plan/apply whereas provider configs need to be known as early as terraform init.
Re:Terraform best practices for reliability at any scale by ftc on Sunday August 06, 2023 @04:10AM (#37019929)
Another option is to use AWS Control Tower: https://docs.aws.amazon.com/controltower/latest/userguide/af...
Re:Terraform best practices for reliability at any scale by denvrede on Saturday August 05, 2023 @02:30AM (#37009525)
How do you deploy your Apps? We exclusively use EKS and having one account per env and app seems like quite an overhead when I think about managing / updating EKS clusters for each one. It also comes with an overhead of base applications that need to run in each cluster by default (like cert-manager, externaldns etc).

Right now we’re using one account per env but also see downsides and thought of going the next step to do one account per env and tribe/team.

Re:Terraform best practices for reliability at any scale by solatic on Saturday August 05, 2023 @08:58AM (#37011579)
GCP and AWS both support sharing a single VPC across multiple GCP projects / AWS accounts. See e.g.: https://aws.amazon.com/blogs/networking-and-content-delivery...

It's not for the faint of heart though. You need to allocate subnets to individual applications (with relevant capacity planning concerns) plus support is sometimes spotty (e.g. EKS doesn't support it last time I checked). Not worth doing until you have several teams trying to use the same VPC and stepping on each others' toes.

Re:Terraform best practices for reliability at any scale by swozey on Saturday August 05, 2023 @12:28PM (#37013646)
Each app/env has a pipeline that will trigger a tf apply in its directory w/ its assumed AWS role and deploy an env after someone gives it a manual approval after looking at the terraform apply/plan output. So it will start at /terrafrom/app1/staging then once healthchecks succeed another manual approval job for /terraform/app1/production will wait to be approved to depoy.

For our EKS apps we do helm rollouts, but most of our services are on ECS so it's mostly just updating a task definition and forcing a redeployment of containers.

Each EKS cluster is set up exactly the same aside from the usual things like vpc and ips and things of that nature that switch between them. They all get a set of "base" apps like log chutes and cert manager and all that as soon as they're deployed.

Our app environments don't communicate with one another at all. The only relationship between them is our IAM accounts in our security account can assume access into them as admin/etc.

Re:Terraform best practices for reliability at any scale by dragonwriter on Friday August 04, 2023 @04:57PM (#37005788)
> But I’m coming at this from a GCP lens and got half way through the article about how the recommended unit of isolation in the AWS environment is entirely different AWS accounts and I’m kind of hung up on that. Is that really a thing people tend to do often? Doesn’t it get super unwieldy?

There are AWS systems above the account level for managing it (Organizations), so its not quite as bad as it might naively seem, but, yes, its more unwieldy than GCP’s projects.

Re:Terraform best practices for reliability at any scale by mdhb on Friday August 04, 2023 @04:59PM (#37005826)
Oh thank god, that’s much better than I naively thought. Thanks for the heads up.
Re:Terraform best practices for reliability at any scale by stock_toaster on Friday August 04, 2023 @04:58PM (#37005791)
You can have sub-accounts that roll up billing to a main account. Still messy, but probably cleaner (security policy wise) and possibly safer (fewer production impacting accidental config changes?) than having a single giant account with _lots_ of things mixed together.
Re:Terraform best practices for reliability at any scale by mdaniel on Friday August 04, 2023 @07:37PM (#37007261)
As the resident pedant, one cannot have "sub-accounts" in AWS. One can 100% have Accounts that are a member of an AWS Organization, which itself has a Management Account that does as you describe as the "main account", but there is no parent-child relationship between Accounts, only OUs and Accounts or OUs and other OUs (which the Organization, itself, counts as an OU)
Re:Terraform best practices for reliability at any scale by stock_toaster on Friday August 04, 2023 @08:29PM (#37007660)
Thanks for the corrections. :)
Re:Terraform best practices for reliability at any scale by lgreiv on Friday August 04, 2023 @04:59PM (#37005822)
You should be able to organize accounts hierarchically using AWS Organizations, which allows to have cost centers and centralized billing (and some imposed policies over all accounts).
Re:Terraform best practices for reliability at any scale by cube2222 on Friday August 04, 2023 @04:59PM (#37005817)
It's extremely common and recommended.

Billing works by having a billing aws account that all other accounts are in a sense "children" of.

Re:Terraform best practices for reliability at any scale by erik_seaberg on Friday August 04, 2023 @07:18PM (#37007123)
Some AWS services have per-account (not per-resource) size and rate limits. Keeping resources in separate AWS accounts gets them out of each others’ blast radii.

AWS IAM doesn’t do capability inheritance; if I can write a policy at all I could grant any privilege to any resource in that policy, even privileges I don’t personally have. It’s easier for each groups of admins to have a separate AWS account than to put everyone in security boundaries that try to wall them off.

Re:Terraform best practices for reliability at any scale by linuxdude314 on Friday August 04, 2023 @07:41PM (#37007306)
This doesn’t have anything to do with AWS, this has to do with terraform not allowing you to dynamically instantiate providers.

The same thing happens with the Kubernetes provider when you try to use it with multiple GKE clusters.