Is a "stack" here a (root) folder on which you'd do a "terraform apply"? I've never know what to call those, surely they aren't "modules".

And, so, you're saying: try to have a separate deployment (stack then?) that contains the state, so you can wipe away everything else if you want to, without having to manage the state?

It's not exactly about the folder, the IaC from a single folder / project can be instantiated in multiple places. Each time you do that, it has a unique state file, so I usually hear it referred to as a "state". In cfn you can similarly deploy the same thing lots of times and each instantiation is called a "stack", so stack/state tend to get used inter-changeably.

And yes, that's a succinct rephrasing.

When you first use iac it maybe seems logical to put your db and app server in the same "thing" (stack or state file) but now that thing is "pet like" and you have to take care of it forever. You can't safely have a "destroy" action in your pipeline as a last resort.

If you put the stateful stuff in a separate stack you can freely modify the things in the stateless one with much less worry.

Can you elaborate on this? I've never heard of this IAC structure and I'm trying to figure out what the benefit/cons are. Maybe it's just Friday and I'm checked out already.

If you run a terraform apply and only update microservices but you also have your dbs/stateful things in the same stack/app, you're only updating the microservices so how would this affect the db/stateful at all?

On the opposite end - I feel like there would be scenarios where I needed to update the stateful AND stateless services with the same terraform apply. Maybe I'm adding a new cluster and adding a db region/replica/securitygroup and that new cluster needs to point at the new db region.

In your scenario I would have updated microservices trying to reach out to a db in a region that doesn't exist yet because I have to terraform apply two different stacks. How would you deal with a depends_on?

Maybe I'm misunderstanding this.

(Hi, I’m one of the authors of the article at the root of this thread.)

Considering your hypothetical stateless microservice change in the same root module as stateful services, problems arise when _someone else_ has merged changes that concern the stateful services, leaving you little room to apply your changes individually.

It’s also worth remembering that, even if a stateless service and a stateful service are managed in the same root module, applying changes is absolutely not atomic. Applying tightly coupled changes to two services “at the same time” is likely to result in brief service interruptions, even if everything returns to normal as soon as the whole changeset is applied.

Ok I think we're talking about two separate things here - you're referencing a root module and not a "stack", as in a stack is a full service/application that uses multiple modules to deploy. Your db module, eks module, etc. All independent modules, not combined into one singular module. Say it's sitting in the /terraform/app1/services/db(&)app folders type of scenario.

I think you're talking about putting stateful and stateless objects inside of a single module. So you've got /terraform/modules/mybigapp/main.tf that has your microservice + database inside of it.

If I'm right and that's what you mean that's really interesting I don't think I've ever seen or done that but now I'm curious. I'm pretty sure I've never created an "app1" module with all of its resources.

Am I totally off here?

It's a complicated conversation because the tools support many different ways of working - and most of them can be made to work with different tradeoffs.

A team that e.g owns its own microservice IaC can absolutely maintain a setup long term with app & db in the same state, it just requires care and love (it's actually easier, it just doesn't scale).

As you split things across more teams or have more complicated IaC "supply chains" (eg teams supplying modules to each other or have lots of ppl working on different bits of the same thing) you need to look at ways to make things more foolproof, easier to support and give yourself as much "wiggle room" as possible for upgrades. At this point having state split out is very helpful (almost essential).

I stuck with my typical term, root module, synonymous with how folks are using “stack” and “state” in various parts of this thread.

A module is any directory with Terraform code in it. A root module is one that has a configuration for how to store and lock Terraform state, providers, and possibly tfvars files. Both modules and root modules may reference other modules. You run `terraform init|plan|apply` in root modules.

I think my comment makes sense in that if you mix two services into the same root module (directly or via any amount of modules-instantiating-other-modules) you can end up with changes from two people affecting both services that you can’t easily sever.

Happy to clarify further if I’m still not addressing your original comment.

@rcowley -- I'm going to preface this with I'm a Staff SRE at an adtech corp that does billions and have been a k8s and terraform contributor since 2015 (k8s 1.1 I forget the tf versions). I don't mean this to brag I just want to set my experience expectation since I'm a random name on hn who you'd never know.

I think calling a service/stack (or whatever, app, etc) a "root module" is a very, very confusing thing to do. Terraform has actual micro objects called modules. We work with them every day. I get how you could consider encompassing an entire chunk of terrafrom code that calls various modules a "root module".. but I think this is just going to lead to absolute confusion to anyone not familiar with your terminology. I don't know every TF conversation but I can't think of a single time where I've heard root module in that context. Very good chance I've just missed those conversation and am ignorant to them.

I'm currently hiring SRE 2s and 3s so I've been interviewing lots of terraform writers and one of my tech questions is to ask someone what makes them to decide to write a terraform module and what type of modules they've written - it's always ALBs, EKS, dbs, etc. components indepedently that go into creating a service/stack. I've definitely not heard anyone mention that they write "root modules" in the sense of an entire service/stack.

I don't mean you're right or wrong, maybe more people are aware of that verbage than I am. I just wanted to mention that in my personal case I think it's confusing so I would assume that there are a lot of people in my shoes who would also be confused by it.

https://developer.hashicorp.com/terraform/language/modules#t... is what Terraform calls the module in the current directory, to distinguish it from child modules you might introduce.

Ok I may absolutely have the dumb today I appreciate the response. The way this is worded, because of this line - "Modules are the main way to package and reuse resource configurations with Terraform." reads like - "I have 10 golang apps, they all at a minimum use these same services, this is our "golang root module". But some services might have more or less modules, ie service A uses redis, service B uses kafka without redis.

So in this verbiage, is every single "stack/app" a "root module" and if one of them has a different database/whatever module it's just using different child modules and the child modules are the big differentiator?

Just to kind of prove the root-module argument I'm making here, this post in here is confused on calling a "stack" a module as well https://news.ycombinator.com/item?id=37005949

Glad we cleared up our terminology! I agree that “root module” risks ambiguity, just like you point out.

I just realized I never responded to the very last point in your original comment. I don’t have, and I don’t think Terraform has, a complete solution to dependencies between root modules. Fortunately, data sources will at least fail if they don’t find what they’re looking for. For me, these failures never come up in production since I’m never using `terraform destroy` there. It does come up in pre-production testing and that’s an area that seems rich with patterns and products on top of Terraform that are controlling order and dependence between root modules.

PS thanks for your work on Terraform and Kubernetes.

Use 'terraform destroy' during CI phase. That is your pre-prod.

Root-module: contains resources, sub-modules, incl. remote module calls.

Stack: deployable with hardcoded tfstate and tfvars configs.

My 2ç as also a (very minor) terraform core, and numerous providers & modules contributor, and user (also 2015 I think): I've never heard of 'terraform stacks' before this thread, but 'root module' makes perfect sense to me:

1) without the context of where the state is/its contents, or estimating based on the resources/style/what's variable vs. hardcoded, a 'stack' (if you like) is indistinguishable from a module that's to be used in someone else's 'stack';

2) `path.root`

lol That's interesting. I feel like all of the TACOS (spacelift, env0, atlantis) refer to stacks.

Thanks for your response It's great to hear corroboration

> it's always ALBs, EKS, dbs, etc. components indepedently that go into creating a service/stack

More importantly, modules represent the DRY principles. We host them in our private Terraform registry and share between teams

Root module is the official terminology used in Hashicorps own documentation. That's actually the term I'm most familiar with in my own experience.

https://developer.hashicorp.com/terraform/language/modules/s...

> Every Terraform configuration has at least one module, known as its root module, which consists of the resources defined in the .tf files in the main working directory.

Not who you replied to, but I was wondering if I could get your take on my use case.

I work on the systems test automation side of things. I use TF to deploy EC2 clusters, including the creation of VPC, subnet, SG, LBs, etc. Once done, I tear down the whole thing.

But from what I'm hearing from yourself and others in this thread, it sounds like I could (/should) be breaking those out separately, even though my use case isn't dev/test/prod(/prod2,/prod3) or even multi-regional.

To rephrase, it sounds like it might be useful for me to be able to create some separation, eg, tear down the EC2 instances in a given VPC/subnet while I ponder what to do next and still leaving the other AWS resources intact, for example. Maybe even deploy another subnet to the same VPC, but a different test intention. I know I can simply specify a different AMI and run tf apply and get one kind of desired deployment change.

Bigger picture: when I need to run one test, I'll copy a dir from our GH repo, edit the tfvars, and kick things off. Another test (in parallel, even), I'll do the same but to a fresh dir. (Wash, rinse, repeat.) And I suspect you're already cringing :) but I get why. It makes me cringe. Ideally I'd be working with a single source of truth (local GH fork).

There's also the consideration of possibly making edits to the stack while I deploy/destroy, while at the same time wanting stability with another deployment for a day or two. I suppose that would require having 2 copies of the remote GH repo. Which is several copies fewer than I'm working with these days.

Fwiw, I've already got "rewrite the stack" on my todo list, but can't get to it for probably another 2 months. So I'm eagerly collecting any "strive to do X, avoid Y like the plague" tips and recommendations prior to diving into that effort.

You’re not wrong. You can achieve the same results using modules well.

It’s common for an entire environment for a whole biz unit or even company to be a single “stack”.

This pretty much only works if the “terraform apply” is centrally orchestrated I.e gitops is the only way to trigger the terraform run.

Agree, it's a complicated conversation because the tools support many different ways of working - and most of them can be made to work with different tradeoffs.

A team that e.g owns its own microservice IaC can absolutely maintain a setup long term with app & db in the same state, it just requires care and love (it's easier in some ways, it just doesn't scale along certain dimensions).

Maybe you have other controls / factors that can make it work for you.

But my experience is that as you split things across more teams or have more complicated IaC "supply chains" (eg teams supplying modules to each other or have lots of ppl working on different bits of the same thing) you need to look at ways to make things more foolproof, easier to support and give yourself as much "wiggle room" as possible for upgrades. At this point having state split out is very helpful (almost essential).

Because the terminology seems to be tripping up the conversation I'd be inclined to phrase it like this: a single "terraform apply" should be touching either a stateful stack, or a stateless stack. Ideally the stateful stacks should be as small as possible, as much infra as possible should be in the stateless stacks.

The way I see it is we only want to use TF for setting up "base infrastructure". This is things like our VPC networks, cloud routers, firewall rules, and finally our Kubernetes clusters.

We still allow devs to use TF, but it's only for using cloud services they depend on, like say a SQL DB or something.

I think you and I are 100% on the same page, but I think the word state (in the way you are using it) is causing a bit of confusion at least for me. A terraform stack will always have state, that's the point. In addition to the current "tfstate" there is a set of parameters that must be used with w/e modules in order to arrive at the tfstate. That's the state that causes problems, not the tfstate so much at least in my experience, as these often are _not_ version controlled.

This is why it's critical to only allow terraform to be applied using automation. I mean don't make it a company policy, make it IAM policy so it's literally not possible.

That's right, stacks can be instantiated across repos even depending upon the organization (both meanings of 'organization' are valid here).

what about prevent_destroy lifecycle?

I have adopted the term "Root Module" vs "Submodule" because those line up with terraform's own definitions, but I agree that they're terribly, terribly named.

I have the same issue. They're all modules, but the ones at the tip of the directory tree (right at the end of your env/region/stack) are called root modules. Which makes no sense because the term "root" always implies that they are at the beginning, not the tippy-toe end. So I call mine "stacks". But as another answer suggested, "states" is also fine. Even though the actual state isn't inside that directory, it's probably in an object store.

At the end of the day I don't care what other people call them.

Makes sense, but how do you connect the two so e.g. credentials from one are surfaced in the other?

Use Data Sources to reference resources in a different state: https://developer.hashicorp.com/terraform/language/data-sour...

Ah - I didn't think about that. Wrapping a remote call with a data source. Thanks.

terraform_remote_state

The root module can have outputs just like any other module. These outputs can be accessed from other stacks from the backend.

And if you use CDKTF the references are handled transparently.

Thanks - another good option! I seem to get slight shivers down my spine when I read terraform_remote_state, but I never actually used it, so I might be half-remembering an old opinionated blog post.

IMO you shouldn't be storing credentials in shared state, as suggested by the other comments, since that means that the principals able to read the state to deploy their service can also read the credentials for other services bundled in that state file. This could be the case if one had broken down the root modules into scopes/services like the linked page suggests.

It is reasonable to assume if you are using Terraform to manage your infra, than your infra likely has access to a secrets manager from your infra vendor, e.g., AWS. Instead I'd recommend using a Terraform data resource to pull a credential from the secret manager by name -- and the name doesn't even necessarily have to be communicated through Terraform state. Then the credentials could directly be fed into where it is needed, e.g., a resource like a Kubernetes Secret. One can even skip this whole thing if the service can use the secret manager api itself. Finally access to the credentials itself would be locked down with IAM/RBAC.

Just stopped here because I had said XYZZY way too often in the last three hours xD

You mean keep stateless separate from stateful?

Everyone else seems to be reading over the typo or I'm more confused than I thought.

Yes

The "stack" nomenclature used here is jarring since it is unrepresented in Terraform HCL literature.

A CDK stack, (assuming that's what is used here), would be loosely equivalent to a Terraform HCL module.

I've never used terraform, but I have used CloudFormation and AWS CDK. It's been a while though, is there a clear indication on the major cloud provider docs which resources are stateful? Or is it always obvious?

Difficult question, as people mean different things when they say state. One example might be a relatively simple AWS Lambda. Most people would say that's easily stateless.

But, what if that Lambda depends on a VPC with some specific networking config to allow it to connect to some partner company private network? And, it's difficult to recreate that VPC without service disruption for a variety of reasons that are out of your control. Well, now you have state because you need to track which existing VPC the Lambda needs if you tear the Lambda down and recreate it.

Usually people use "state" to mean that the resource accumulates data. E.g., a VPC is stateless, but the s3 bucket containing flow logs for the VPC is not.

Tearing down and recreating a VPC can be disruptive, but that doesn't make it a stateful resource.

In my example, the "state" part is someone storing the specific VPC reference to re-apply to a newly re-created lambda. Rather than recreating both the VPC and the Lambda. Many people would refer to that as a bad practice because it's carrying state. e.g., the kind of stuff that terraform puts in .tfstate files.

There's definitely a purist camp that advocates completely stateless infra. Though there's also what you're describing, where some people consider state to only refer to things like databases.