We started out with just one terraform project per account. Now its a terraform project per project within an account, to keep terraform projects compact.
But it somehow keeps growing and growing and each team and each project, they handle it differently. the taste is differently, it seems. and not everyone puts a lot of thought into rhe structure. So on the one hand its wild and free, but on the other i palm my face, when i look into the future.
I’ve been at multiple companies where we’ve decided to build a set of modules that could be used somewhat universally for products, and then we’d basically say “hey if you don’t want to use this, either convince us that your needs are justified or do whatever you want and DevOps/SRE can’t support you.” Some of it we went as far as calling self-service.
If you have a menu of options than people can’t pick things that aren’t on the menu.
I also find that developers really just shouldn’t have almost any cloud infrastructure rights, with few exceptions.
You could also go the embedded SRE/DevOps route where the embedded infrastructure engineer on the product team can steer each separate product into following more standard approaches.
I’m also going to take a wild guess that your ops to developer ratio is too low.
“We only have 3 menu options of ways to do things” is great for creating hierarchies, and rarely works for actually creating any innovation.
Are you leveraging modules? For example, if you create a new product in your account, can you just have the product call your own module to create its IAM roles to your own standards and specification? Or are you repeating IAM code over and over?
Basically all of your terraform code should be modularized because you’ll likely repeat it for different environments like staging vs. production. So, for example, when I want to build another product VPC in another environment, it’s just another deployment of the same child module with different input variables.
Are you trying to manage multiple products with the same terraform code? I see this a lot and while it’s not a catastrophe, I try to separate terraform repositories by their product or logical function.
I also wonder if you’re overusing AWS sub-accounts. I only use a separate account when the business needs to separate the business unit entirely, or to separate production from other environments. Everything else should be able to be kept separate via roles and policies.