News for hackers. Stuff that matters.
Re:Terraform best practices for reliability at any scale by rcarr on Friday August 04, 2023 @08:47PM (#37007769)
Genuine question for DevOps people:

Other than the fact it seems to be an industry standard so it's good for your job prospects, what are the benefits to Terraform over CloudFormation/CDK or whatever the equivalent is for your particular cloud provider?

Most companies/people pick a provider and then stick with it and it doesn't seem like there's much portability between configurations if you do decide to switch providers later down the line so I'm not sure what the benefits are. I haven't delved into Terraform yet but I tried doing a project in Pulumi once and felt by the end of it that I might as well have just wrote it in AWS CDK directly.

Re:Terraform best practices for reliability at any scale by Centigonal on Friday August 04, 2023 @09:00PM (#37007851)
I worked closely with the folks that wrote our platform's IaC, first in CDK, then in Terraform. I wrote a bit of CDK and zero TF myself, but here are some of the reasons we switched:

A big plus is that Terraform works outside of AWS land.

CDK is a nightmare to work with. You're writing with programming-language syntax, which tempts you to write dynamic stuff - but everything still compiles down to declarative CFN, which just makes the ergonomics feel limited. The L2 and L3 constructs have a lot of implicit defaults that came back to bite us later.

With CDK you get synth and deploy, which felt like a black box. Minor changes would do the same 8 minute long deploy process as large infrastructure refactors. Switching to TF significantly sped up our builds for minor commits. There might be a better way to do this with CDK (maybe deploying separate apps for each part of our infrastructure) and we may have just missed it.

Re:Terraform best practices for reliability at any scale by jahsome on Friday August 04, 2023 @08:53PM (#37007806)
Third-party integrations and the universality/reusability across multiple products and familiarity of HCL are big for me.
Re:Terraform best practices for reliability at any scale by thedougd on Friday August 04, 2023 @09:00PM (#37007852)
Providers I regularly use, even mixed in a single project. There are others I could use if they were available.

AWS GitHub Opsgenie Okta Scalr TLS DNS

Re:Terraform best practices for reliability at any scale by koolba on Friday August 04, 2023 @09:02PM (#37007871)
You forgot the greatest escape hatch of all: null
Re:Terraform best practices for reliability at any scale by x3n0ph3n3 on Friday August 04, 2023 @08:54PM (#37007816)
I have used both terraform and cloudformation substantially and they each have pros and cons. One thing terraform has over cloudformation is its rapid support for new services and features. AWS has done an awful job ensuring that cloudformation support is part of each team's definition of "done" for each release. It just doesn't get the support it really needs from AWS.
Re:Terraform best practices for reliability at any scale by koolba on Friday August 04, 2023 @09:01PM (#37007860)
CloudFormation is the ugly step child of AWS. It has bugs that have languished for years
Re:Terraform best practices for reliability at any scale by viraptor on Saturday August 05, 2023 @10:13AM (#37012191)
Or weird missing functionality like some resources being impossible to tag from the stacks, or new services/options lagging months behind when TF implemented support for them.
Re:Terraform best practices for reliability at any scale by yellowapple on Friday August 04, 2023 @09:45PM (#37008152)
- In the event that you are working with different cloud providers, Terraform is one thing to learn that then applies to all of them, as opposed to learning each provider's bespoke infra-as-code offering. Most companies stick to one PaaS/IaaS, but individual personnel ain't necessarily as limited over the courses of their careers.

- Not all cloud providers have an infra-as-code offering of their own in the first place (especially true with traditional server hosts), whereas pretty much every provider with some sort of API most likely has a Terraform provider implemented for it.

- Terraform providers include more than just PaaS/IaaS providers / server hosts; for example, my current job includes provisioning Datadog metrics and PagerDuty alerts alongside applications' AWS infra in the same per-app Terraform codebase, and a previous job entailed configuring Keycloak instances/realms/roles/etc. via Terraform.

Re:Terraform best practices for reliability at any scale by androidbishop on Saturday August 05, 2023 @02:00AM (#37009388)
Also pretty neat that there's a Terraform provider for Kubernetes native resources.
Re:Terraform best practices for reliability at any scale by mcfedr on Saturday August 05, 2023 @07:35AM (#37011015)
Personally it's the only way to work with kubernetes. Terraforms change control is excellent. Cannot go back to helms upgrade and how it does what you want.
Re:Terraform best practices for reliability at any scale by dgrin91 on Friday August 04, 2023 @10:03PM (#37008259)
Companies choose providers and tend to stick with them, but people don't always stick with companies. If I know TF there is a decent chance my skills will be applicable when I change companies.

Also some big corps run their own internal datacenters and have cloud-like interfaces with them. You can write TF providers for that (its not going to be as nice as the public cloud ones, but still nice). Then you can utilize Terraforms multi-provider functionality to have 1 project manage deployments on multiple clouds that include on-prem.

Also terraforms multi-provider functionality is also useful for non aws/azure/gcp such as Cloudflare. As far as I know CDK does not support that.

Re:Terraform best practices for reliability at any scale by RulerOf on Friday August 04, 2023 @10:54PM (#37008515)
I've got a lot of opinions here, but the only one I'll share is that HCL knocks the socks off of json and yaml. Json is too rigid. YAML is too nested. HCL gets this just right.

Venturing away from opinions, the provider ecosystem with terraform enables some wonderful design options. For example, I have a module template that takes some basic container configs (e.g. ports, healthchecks) and a GitHub URL, then stands the service up on ECS and configures CI in the linked repo. CF can't do that.

Re:Terraform best practices for reliability at any scale by abrookewood on Friday August 04, 2023 @11:17PM (#37008644)
After you have waited 20 minutes for CloudFormation to fail and tell you that it can't delete a resource (but won't tell you why), and this is the third time it has happened in a week, you start looking at alternatives.
Re:Terraform best practices for reliability at any scale by androidbishop on Saturday August 05, 2023 @01:57AM (#37009379)
This 1000%. Also recently discovered Google Deployment Manager is shit for the exact same reasons. I honestly don't get it.
Re:Terraform best practices for reliability at any scale by 333throwaway342 on Friday August 04, 2023 @11:32PM (#37008739)
> Most companies/people pick a provider and then stick with it and it doesn't seem like there's much portability between configurations if you do decide to switch providers later down the line so I'm not sure what the benefits are.

This smells like kubernetes

> Terraform over CloudFormation/CDK

They both work. It's more about which providers you need.

Re:Terraform best practices for reliability at any scale by Illotus on Saturday August 05, 2023 @12:27AM (#37009020)
> Other than the fact it seems to be an industry standard so it's good for your job prospects, what are the benefits to Terraform over CloudFormation/CDK or whatever the equivalent is for your particular cloud provider?

For me the killer feature is that both plan and apply show the actual diff of changes vs running infrastructure. It makes understanding effects of changes much easier.

Re:Terraform best practices for reliability at any scale by Bellyache5 on Saturday August 05, 2023 @08:50AM (#37011518)
Agreed, Terraform does a good job of this. But CloudFormation & CDK can also do this via Change Sets and CDK diff.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

https://blog.mikaeels.com/what-does-the-aws-cdk-diff-command...

Re:Terraform best practices for reliability at any scale by nuker on Saturday August 05, 2023 @01:24AM (#37009250)
Im 10 years working with AWS. I strongly prefer Cloudformation, just separate things smartly between stacks. It has export/import for stack outputs too. Just look at the “root module” mess in this discussion and you’ll get why.
Re:Terraform best practices for reliability at any scale by androidbishop on Saturday August 05, 2023 @01:56AM (#37009376)
Terraform, and by extension HCL, is more powerful and flexible. It can be used across clouds. It has providers for all kinds of things, like kubernetes. It can be abstracted and modularized. It supports cool features like workspaces and junk, depending on how you want to use it.

Also recently I was forced to use Google Cloud Deployment Manager scripts for some legacy project we were migrating to Terraform, and I was shocked at how buggy and useless it was. Failed to create resources for no discernible reason, couldn't update existing resources with dependencies, couldn't delete resources, was just unfathomably shit all around. Finished the Terraform migration earlier this morning and everything went off without a hitch, plus we got more coverage for stuff Deployment Manager doesn't support. It's also organized much nicer now, with versioned modules and what-have-you.

Cloudformation is ugly and again, surprisingly isn't well supported by AWS. I don't understand how it's possible, but terraform providers seem to be more up to date with products and APIs. Maybe that's just me but I've seen others complain about the same thing.

Re:Terraform best practices for reliability at any scale by wodenokoto on Saturday August 05, 2023 @02:14AM (#37009457)
Isn’t google cloud deployment just bash calls to the google cloud cli disguised as declarations by way of yaml?
Re:Terraform best practices for reliability at any scale by androidbishop on Saturday August 05, 2023 @03:48AM (#37009921)
Considering the fact that you can inject python code blocks, I kind of doubt it. It also makes API calls that populates a dashboard with an inventory of resources created, so it seems to be more of an api wrapper like other IaC solutions.
Re:Terraform best practices for reliability at any scale by cwp on Saturday August 05, 2023 @04:09AM (#37010028)
It's subtle and so difficult to see the differences at smaller scales. If you're going to provision a handful of EC2 instances, all the tools work fine.

I think HCL is an under appreciated aspect of Terraform. It was kinda awful for a while, but it's gotten a lot better and much easier to work with. It hits a sweet spot between data languages like JSON and YAML and fully-general programming languages like Python.

Take CloudFormation. The "native" language is JSON, and they've added YAML support for better ergonomics. But JSON is just not expressive enough. You end up with "pseudoparameters" and "function calls" layered on top. Attribute names doubling as type declarations, deeply nested layers of structure and incredible amounts of repetitious complexity just to be able to express all the details need to handle even moderate amounts of infrastructure.

So, ok, AWS recognizes this and they provide CDK so you can wring out all the repetion using a real programming language - pick your favourite one, a bunch are supported. That helps some, but now you've got the worst of both worlds. It's not "just JSON" anymore. You need a full programming environment. The CDK, let's say the Python version, has to run on the right interpreter. It has a lot of package dependencies, and you'll probably want to run it in a virtualenv, or maybe a container. And it's got the full power of Python, so you might have sources of non-determinism that give you subtle errors and bugs. Maybe it's daylight saving gotchas or hidden dependencies on data that it pulls in from the net. This can sound paranoid, but these things do start to bite if you have enough scale and enough time.

And then, all that Python code is just a front end to the JSON, so you get some insulation from it, but sometimes you're going to have to reason about the JSON it's producing.

HCL, despite its warts, avoids the problems with these extremes. It's enough of a programming language that you can just use named, typed variables to deal with configuration, instead of all the { "Fn::GetAtt" : ["ObjectName", "AttName"] } nonsense that CloudFormation will put you through. And the ability to create modules that can call each other is sooo important for wringing out all the repetition that these configurations seem to generate.

On the other hand, it's not fully general, so you don't have to deal with things like loops, recursion, and so on. This lack of power in the language enables more power in the tools. Things like the plan/apply distinction, automatically tracking dependencies between resources, targeting specific resources, move blocks etc. would be difficult or impossible with a language as powerful as Python.

HCL isn't the only language in this space - see CUE and Dhall, for example - but it's undoubtedly the most widely used. And it makes a real difference in practice.

Re:Terraform best practices for reliability at any scale by solatic on Saturday August 05, 2023 @05:26AM (#37010395)
> your particular cloud provider

Just this week, I wrote a Terraform module that uses the GCP, Kubernetes, and Cloudflare providers to allow us to bring up a single business-need (that will be needed, hopefully, many times in the future) that spans those three layers of the stack. 200 lines of Terraform written in an afternoon replaced a janky 2,000 line over-engineered Python microservice (including much retry and failure-handling logic that Terraform gives you for free) whose original author (not DevOps) moved on to better pastures.

CDK is fine if you're all-in on AWS. It has its tradeoffs compared to Terraform. Pick the right tool for the job.

Re:Terraform best practices for reliability at any scale by maccard on Saturday August 05, 2023 @07:53AM (#37011116)
We don't just have AWS resources. Our CI pipelines are managed by terraform [0], they communicate with GitHub [1]. I like that it's declarative and limited, it stops people trying to do "clever shit" with our infra, which is complicated enough as it is.

[0] https://buildkite.com/blog/manage-your-ci-cd-resources-as-co...

[1] https://registry.terraform.io/providers/integrations/github/...

Re:Terraform best practices for reliability at any scale by raffraffraff on Saturday August 05, 2023 @08:07AM (#37011212)
For me personally, I chose terraform because it can work with AWS and a heap of other 3rd party services and software (Cloudflare, PostgreSQL, Keycloak, Kubernetes/Helm, Github, Azure)