What if terraform was able to perform a complete audit of your environment for each provider and say "this is what you've got, mate", then give you options for each resource:
1. Accept (turns it into terraform code)
2. Reject (removes the resource)
Taken further, this idea effectively means that manually creating something like EKS could be a way to auto-compose Terraform code and store it in git.
Pros:
* You have the options of writing code, using the console or using a script or binary to perform a complex setup
* You have 100% code coverage once Terraform compared code with reality and forced you to choose
Cons:
* Auto-generated code is shitty.
* Slooooooow without caching (which is only one of the functions of state)
To make these concepts work without a remote state you pretty much have to run your infrastructure tool as a service, and implement things like:
1. Polling of providers on an interval
2. Alerting when something changes
3. Eventually consistent cache that can be forcibly updated in the UI or during sensitive operations
4. "Infrastructure Composer" UI, like a vastly simplified IaaS console that only shows deployed infrastructure and lets you group infrastructure by service, and let you define code-level modules
5. Show discovered dependencies and ordering, and allow you to explicitly set them (these should be stored as tags on the actual infrastructure components where possible)
6. Use some hot AI to group code-level resources, compare their configurations and generate config data to apply to resources/modules using for_each loops, while respecting service boundaries and offering advice on when to refactor.
We keep coming up with these layers of abstractions around stuff that at the bottom is essentially already stateful but just not in a useful way. Doing it right from the ground up might help. Of course, left to the usual suspects, this would just turn into another Frankenstein blackhole of devops time. The usual suspects are billion dollar corporations that thrive on layers of complexity.
This is how we do our initial terraform for new projects. We build up what is missing by hand, then export the terraform to save time and effort in building our own internal templates. We throw away most of it, but it is very useful for not having to look up how every aspect is named and what options are needed.
0: https://docs.public.oneportal.content.oci.oraclecloud.com/en...
Edit: and actually, without a state there is no concept of a remote state. All codebases talk to the same "service" so circling a bunch of infrastructure and saving it to a specific git project would be easier than connecting to a bunch of remote states and reading specific resource days from them. Even if you're code repo doesn't declare a resource, the omniscient IAC service "knows all"
I believe, Terraform needs a mode that removes unmanaged resources, but that's quite the big ask and would result in yet another API / SDK change that providers would need to implement.
Measured cloud setup would have VMs A + X + Y
New cloud setup would specify VMs B + X + Y
You can easily identity X and Y as their names go unchanged, A and B would have similar config/metadata, instead of assuming A would be added and B would be removed you can ask the user if a rename happened.
This is not a new problem to solve by the way, this is how database migration tools handle column renames.
In most places I've been, Terraform is scattered, each run managing its own corner of the infrastructure. In this case, each Terraform run would delete everybody else's infrastructure.
Also, how do you know when to re-create things like random strings or numbers? Or null resources? Presently it's a mix of keepers (requires state) and taint (also requires state).
Terraform would be much better without state. Not 10x better, but 2x.
Orphan resources only exist as a concept if you manage entire estates in a single configurations, which is simplistic to the extreme.
If I use local_file to emit a certificate authority for an EKS cluster, does that mean every other file on my file system is suddenly orphaned? What if the networking team responsible for VPCs, routing and transit search for orphaned resources in my AWS account - should that include all my instances?
If you think Terraform would be better without state, I’d encourage you to put your money where your mouth is and build it.
When you change the name of the resource that will, obviously, break the connection, that's a big no-no when writing Terraform code.
One problem, as mentioned in a sibling comment, is if you don't have state then how do you know a resource has been deleted? One other option might be a system that compares the "desired state" from the previous version of the config. That might be an interesting approach - keeping a history of config. But that is, of course, state :)
I use TF a bit. I want to like it, but I do spend inordinate amounts of time faffing around with state files to make them match reality, and I'm not even doing particularly complex stuff. I'm told other tooling (eg: Bicep for Azure) dispenses with state entirely.
Bicep/ARM still has issues with resource deletion, though. The default deployment behavior is to ignore resources that aren't described in a deployment template, so if you remove a VM from your template and redeploy, the VM will keep running until you manually delete it. There are a couple ways around this issue, but they all rely on having state external to the resource itself.
Disclaimer: I work on Bicep/ARM and think it's pretty great, but it's not perfect.
Regarding the state, it took me a very long time to come up with good organization for my code, but it works once you've gotten used to it. I really only need to mess with state files when there is a bug in the provider like the aforementioned GitHub issue.
The whole point of TF is that it has state and doesn't require workarounds for these scenarios. Yes, you have to maintain state, but the state problems usually come from buggy providers, not Terraform itself. For example, try and use the GitHub provider to create a repo with GitHub Pages enabled from the gh-pages branch. It won't work because the authors didn't respect a fundamental rule of writing a provider: one state-changing API call = one resource. If you don't respect that, you have to do state handling yourself and you're almost certain to have state-related bugs.