News for hackers. Stuff that matters.
Re:Infrastructure as code is not the answer (2022) by pquki4 on Friday April 05, 2024 @08:08AM (#39941381)
The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't address any of these.

> I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late.

That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.

Re:Infrastructure as code is not the answer (2022) by jpdb on Friday April 05, 2024 @08:21AM (#39941506)
And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs.

You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.

Re:Infrastructure as code is not the answer (2022) by zaphar on Friday April 05, 2024 @08:47AM (#39941733)
The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production.

Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.

This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.

Re:Infrastructure as code is not the answer (2022) by zzyzxd on Friday April 05, 2024 @03:54PM (#39946464)
GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race).

To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.

If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.

It should have never been called infra-as-code, but infra-as-data.

See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/

Re:Infrastructure as code is not the answer (2022) by zaphar on Friday April 05, 2024 @10:44PM (#39949577)
This does not change my statement at all though? You fundamentally can't really predict the impact of some changes in a given environment until it's deployed. Just because you can obtain the current state of the environment and reconcile some stuff doesn't change this.
Re:Infrastructure as code is not the answer (2022) by zzyzxd on Saturday April 06, 2024 @02:33AM (#39950476)
That's why you should call what you store in git the _desired_ state, not anything else. A git repository is not a live database. It's a collection of static text files that change less often than your live system. There will be bugs and misconfiguration, and sometimes the desired state is just technically not reachable, and that's fine. What the actual state is doesn't matter. Leave that to the controller. State drifting is a problem your gitops engine should detect, and should be fixed by the owner of controller code.

Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.

Re:Infrastructure as code is not the answer (2022) by zaphar on Monday April 08, 2024 @10:20AM (#39970004)
This is correct. You need some kind of running check on the environment and when possible code that handle exceptional cases.

Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".

Re:Infrastructure as code is not the answer (2022) by jansommer on Friday April 05, 2024 @08:28AM (#39941561)
Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
Re:Infrastructure as code is not the answer (2022) by thomascgalvin on Friday April 05, 2024 @08:28AM (#39941557)
This article is basically one long "be careful, that hammer can hit your thumb."

While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative.

Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.

Re:Infrastructure as code is not the answer (2022) by nonethewiser on Friday April 05, 2024 @08:59AM (#39941820)
It doesnt absolve the whole article but a delete command for your entire infrastructure is obviously a major footgun that doesnt exist with alternatives (to use one of your own lenses).
Re:Infrastructure as code is not the answer (2022) by jolux on Friday April 05, 2024 @09:09AM (#39941911)
It doesn't exist with Terraform either if you are following least privilege and using a cloud provider.
Re:Infrastructure as code is not the answer (2022) by pksebben on Friday April 05, 2024 @10:02AM (#39942496)
Or if you just use the terraform cli properly and actually glance at the report it gives you before the “hey guy are you really, really sure about this?” Safety prompt.

You not paying attention is not terraforms fault.

Re:Infrastructure as code is not the answer (2022) by adwf on Friday April 05, 2024 @10:09AM (#39942568)
Eh, Azure has a big delete button for any Subscription. Seen it happen and take down a production system.

Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?

Re:Infrastructure as code is not the answer (2022) by kthejoker2 on Sunday April 07, 2024 @08:52AM (#39960418)
To be fair when you click the button:

* it throws a.big warning text

* it lists out all the resources you're deleting

* you have to type out the name of the subscription

* click another button

* and then another yes I'm sure button with more warning text

At the end of the day they have to provide a button.

Re:Infrastructure as code is not the answer (2022) by groestl on Friday April 05, 2024 @09:06AM (#39941883)
"It takes an admin to take down a server, it takes a configuration management system to take down a site".
Re:Infrastructure as code is not the answer (2022) by zmgsabst on Friday April 05, 2024 @09:00AM (#39941829)
In particular, IaC replaced “thousands of lines of YAML and regex” or “magic shell script” solutions.

IaC is much better than what it replaced, even if there are still many sharp corners.

Re:Infrastructure as code is not the answer (2022) by cjonas on Friday April 05, 2024 @09:01AM (#39941843)
The "as code" means the quality and stability of the implementation varies wildly. If you use a declarative style with proper SDLC in place, then it's fantastic.

On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.

Re:Infrastructure as code is not the answer (2022) by SideburnsOfDoom on Friday April 05, 2024 @09:04AM (#39941862)
> If you do 'DROP TABLE' in production, nothing can save you.

Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'

Re:Infrastructure as code is not the answer (2022) by nojvek on Saturday April 06, 2024 @06:10AM (#39951357)
That’s not infrastructure as code means. It means engineers wouldn’t have ability to run terraform on laptop at all. They have to make a PR, get it reviewed and the code lands in some prod level branch where CI would run terraform for them.

And if things got messed up, revert the commit and we are back (mostly - May need backup restores).

It means your infrastructure is a commit that you can see history and forward/reverse.

Git represents your infra. If it’s not in the repo, it’s not there.