You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.
This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.
If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.
It should have never been called infra-as-code, but infra-as-data.
See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/
Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".
While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative.
Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.
You not paying attention is not terraforms fault.
Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
* it throws a.big warning text
* it lists out all the resources you're deleting
* you have to type out the name of the subscription
* click another button
* and then another yes I'm sure button with more warning text
At the end of the day they have to provide a button.
IaC is much better than what it replaced, even if there are still many sharp corners.
On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.
Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'
And if things got messed up, revert the commit and we are back (mostly - May need backup restores).
It means your infrastructure is a commit that you can see history and forward/reverse.
Git represents your infra. If it’s not in the repo, it’s not there.
> I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late.
That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.