News for hackers. Stuff that matters.
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @09:13AM (#27434695)
I strongly agree both with respect for the half-baked-ness of the language and with the "it's probably the best out there". Ultimately, these tools should have a static/yaml-like "assembly language" that describes the state of your infrastructure without any of the DRY. There would be a diffing engine which would figure out what changes need to be applied and apply them accordingly. Users could use some vanilla programming language to generate that yaml in a DRY way; then the Terraform folks don't need to badly reinvent a programming language.

I know they also have a CDK, but I can't tell if it properly solves that problem or if it still forces us into Terraform idiosyncrasies (i.e., if I rename something in Terraform, it will try to delete the corresponding resource and recreate it, and I think that absurd behavior remains with the CDK).

Re:Terraform 1.0 by kevincox on Tuesday June 08, 2021 @09:32AM (#27434935)
100%. Terraform is half-way between a tool for generating the configuration and applying it. I think Terraform's application engine is actually quite good, but I would like to use a much better tool to generate the config. (And be able to diff that config)

You can feed JSON to Terraform however this falls over if you need dependencies for output values. This usually isn't an issue because most Cloud provider resources have predictable IDs but as soon as you have one that doesn't you are up for a lot of pain and suffering.

Re:Terraform 1.0 by ekimekim on Tuesday June 08, 2021 @10:50AM (#27435927)
You may be interested in Pulumi: https://www.pulumi.com/

Basically it's Terraform but instead of declaring your resources in HCL, you declare them in a real programming language. You're still producing a declarative config that the engine then diffs, applies etc. In fact, it's compatible with existing terraform providers, so it has a surprisingly large selection of things you can use it for.

Note their docs will try to guide you towards using their hosted service which basically does nothing except host the state file, but you can use an S3 or GCS bucket instead and it works fine.

It's definitely not without its own problems, but I'd say it's overall an improvement.

Re:Terraform 1.0 by zinclozenge on Tuesday June 08, 2021 @01:52PM (#27438534)
Unfortunately last I checked, pulumi only offers state locking with their paid service. If you want to self-host you have to implement it yourself, which seems like a non-starter for a lot of people.
Re:Terraform 1.0 by lmzen on Tuesday June 08, 2021 @04:01PM (#27440023)
This was addressed a couple months ago in https://github.com/pulumi/pulumi/pull/2697
Re:Terraform 1.0 by zinclozenge on Wednesday June 09, 2021 @03:43PM (#27452205)
Wow it took 2 years for the PR to get merged.
Re:Terraform 1.0 by deadbunny on Tuesday June 08, 2021 @02:42PM (#27439166)
Re:Terraform 1.0 by arcticfox on Tuesday June 08, 2021 @11:17AM (#27436315)
Glad somebody mentioned Pulumi. It solved all of the major problems I had with Terraform.
Re:Terraform 1.0 by nprateem on Tuesday June 08, 2021 @11:59AM (#27436880)
Not with that licensing thanks
Re:Terraform 1.0 by simcop2387 on Tuesday June 08, 2021 @12:15PM (#27437147)
It looks like it's Apache 2.0 licensed? Wh issues do you have with that licemse.
Re:Terraform 1.0 by renewiltord on Tuesday June 08, 2021 @12:11PM (#27437084)
It’s Apache 2, isn’t it? What’s wrong with that?
Re:Terraform 1.0 by mwarkentin on Tuesday June 08, 2021 @09:48AM (#27435162)
Re:Terraform 1.0 by cormacrelf on Tuesday June 08, 2021 @10:52AM (#27435959)
Someone should make a Clojure demo of those Java bindings, or even cljs. I hope Clojure has good type based completions these days, because it would be a fantastic language for this.
Re:Terraform 1.0 by cormacrelf on Tuesday June 08, 2021 @10:48AM (#27435891)
It’s pretty wild that the object identity via name thing is still a problem. Can they not add a transitional name feature where an object is known by multiple aliases for a while and then when you have finished putting though a change, you can delete the original name? Is this not very basic SQL migration practice? Like column aliases until no longer needed.
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @11:39AM (#27436599)
I don't even understand why the state needs to know the identifiers that the high level language uses for various resources. If the high level language has a binding "foo_bucket" for an AWS S3 bucket resource with a single property `name = "foo"`, then why should the state need to know that the high level language refers to that bucket with the name "foo_bucket"? Instead, the state should look something like this (obviously simplified):

    {
        "resources": [
            {
                "type": "aws_s3_bucket",
                "properties": {"name": "foo"}
            }
        ]
    }
Note that there is no reference to "foo_bucket".
Re:Terraform 1.0 by devonbleak on Tuesday June 08, 2021 @02:05PM (#27438695)
This doesn't make sense to me. You need to know the logical identifier in order to explicitly link the code with the resource. Otherwise if I change the code for that resource how does TF know what it needs to change if none of the existing resources in state matches the new config? Do you just always destroy and re-create every time there's a change to anything?
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @03:07PM (#27439457)
> Otherwise if I change the code for that resource how does TF know what it needs to change if none of the existing resources in state matches the new config?

A resource provider defines a collection of fields that is the "identifier" for the resource. For example, an S3 bucket resource would have the "name" field for its identifier.

If you change another attribute besides the bucket name, the engine will see that the input and the state both have a s3 bucket resource with the same name but different props, so it knows it will need to update some props (rather than create a new one). However, if the name changes, the engine will see that the input has a bucket that doesn't exist in the state so it will add a "create bucket" step to the plan. It will also see that the state has a bucket that isn't in the input, so it will add a "delete bucket" step to the plan.

Maybe another way of saying the same thing is that a resource provider can mark any given field as "forces replacement", and all of the fields that force replacement are the de facto identifiers? I haven't thought through whether these are exactly equivalent.

Re:Terraform 1.0 by devonbleak on Tuesday June 08, 2021 @06:39PM (#27442022)
The "identifier" is often something that's computed later or returned from the API. Think about something like an ec2 instance - the identifier is the instance ID that's returned from AWS. You can have many instances that basically look identical so how do you differentiate which one this logical resource is referencing?

And back to the s3 bucket use case sometimes you want uniqueness in your name so you use a prefix instead of specifying the whole name - how do you determine which bucket that resources is referencing if there are multiple buckets matching the prefix?

I hear what you're saying in terms of wanting state management to be simplified, but pretty much every IaC solution uses this explicit logical resource -> physical resource mapping in state.

Re:Terraform 1.0 by kevincox on Tuesday June 08, 2021 @11:04AM (#27436141)
Yeah, moving objects around the config is common if you want to keep it organized and requires manual actions that require essentially a global lock on the stack (and Terraform has no built-in feature to actually take this lock). It makes it basically impossible to implement a fully automated production change pipeline with Terraform.
Re:Terraform 1.0 by cormacrelf on Tuesday June 08, 2021 @11:54PM (#27444046)
Moreover I can never, ever, remember the syntax for moving objects around the config. It's really painful.

Edit: the aliases would have to handle moving as well as renaming. You could just have aliases in a global namespace, which means adding `alias = "portable-elb"` and doing one `terraform apply` means you can pick up that config, drop it anywhere else, and it will move it for you. It wouldn't even need to do a full `apply`, just a local JSON manipulation.

Re:Terraform 1.0 by ethbr0 on Tuesday June 08, 2021 @12:02PM (#27436927)
> application engine [vs] tool to generate the config

I get it from HashiCorp's perspective though.

A robust application engine with a suboptimal config generator is a viable product.

A suboptimal application engine with a brilliant config generator is not.

So given limited resources, former gets the dev grease.

Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @02:08PM (#27438735)
This is a false dichotomy.

You can generate these configs really easily with any off-the-shelf programming language for a small fraction of the effort they’ve put into HCL + all of the stuff on top that makes HCL the shitty programming language that it is.

Even if you insist on building your own programming language for this purpose, Hashicorp could’ve saved themselves a lot of work by looking at the prior art of the last 70 years of programming language history.

In other words, if they just picked, say, JavaScript from the start they could have saved a bunch of time and energy and put that into their application engine.

Re:Terraform 1.0 by lvncelot on Tuesday June 08, 2021 @10:42AM (#27435813)
> You can feed JSON to Terraform however this falls over if you need dependencies for output values

This is what I've started doing with Jsonnet for generation, and also exactly why I've stopped doing it.

Re:Terraform 1.0 by jen20 on Tuesday June 08, 2021 @12:23PM (#27437269)
I'm not sure I follow exactly what you're missing. `${aws_instance.example.x}` as a string value creates the same dependency as it would via HCL when used with JSON.
Re:Terraform 1.0 by polynomial on Wednesday June 09, 2021 @03:23AM (#27445187)
Same here, I don't see how outputs is being treated any differently by Terraform than any other .tf file written in HCL. I'm not saying it's not possible, but I haven't experienced a failure more there yet.
Re:Terraform 1.0 by lvncelot on Thursday June 10, 2021 @03:12AM (#27457809)
Thanks for the hint, now I'm not sure what went wrong when I tried something like this. I should read up on this more.
Re:Terraform 1.0 by polynomial on Tuesday June 08, 2021 @01:10PM (#27437916)
What are some of the tools that do this? The only ones I know of are Scalr and Pulumi.
Re:Terraform 1.0 by nuker on Tuesday June 08, 2021 @09:38AM (#27435013)
> Ultimately, these tools should have a static/yaml-like "assembly language" that describes the state of your infrastructure without any of the DRY.

CloudFormation ?

> There would be a diffing engine which would figure out what changes need to be applied and apply them accordingly.

CloudFormation.

Re:Terraform 1.0 by oneplane on Tuesday June 08, 2021 @10:03AM (#27435343)
Problem with CloudFormation is that it doesn't work with Cloudflare, Azure, GCP, Big-IP, Palo Alto, NetBox etc..
Re:Terraform 1.0 by nuker on Tuesday June 08, 2021 @10:13AM (#27435457)
Its a problem only if you use these vendors, you don't have to.
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @10:22AM (#27435559)
It's a pretty tough sell to tell people they have to uproot all of their existing infrastructure and move to Amazon just to use an infra-as-code tool.
Re:Terraform 1.0 by oneplane on Tuesday June 08, 2021 @02:24PM (#27438910)
It's also unlikely that you will only use AWS, forever. At some point in time you'll have to deal with various resources (be it IT resources, time, money or people-as-a-resource), and whenever you bind your knowledge and workforce to an IaC tool that doesn't transfer or isn't portable you're going to end up with N+1 tools every time. In other words: it doesn't scale all that well. (And that doesn't mean Google-scale, but going from 2 IaC engineers to 5 IaC engineers is much harder if you can't apply universal tooling)

Tools are never 'just tools', there is context and there are externalities. And as you already pointed out: migrating/uprooting all of those other things isn't a likely scenario.

Re:Terraform 1.0 by throwaway894345 on Wednesday June 09, 2021 @09:34AM (#27447660)
Agreed. If you use an auth service (SaaS or self-hosted) that isn't AWS Cognito you will also find yourself wanting to integrate with your IaC tool. Having to roll this yourself with CloudFormation is a lot of effort, or at least it was last time I looked, and importing a third party "provider" wasn't really a thing.
Re:Terraform 1.0 by zymhan on Tuesday June 08, 2021 @10:17AM (#27435505)
Fun fact: You don't even have to use Terraform
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @09:51AM (#27435188)
Yeah, CloudFormation is workable in this regard (I've created a neat generator for Python), although it has lots of its own problems (e.g., if you want to create a new resource, you have to run it as its own lambda--your infra-as-code needs its own infra which needs its own infra-as-code).
Re:Terraform 1.0 by polynomial on Tuesday June 08, 2021 @01:54PM (#27438559)
> I've created a neat generator for Python

care to share? (I know some hn users often don't w/o being asked, out of a sense of not wanting to be seen as self-promoting.)

Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @02:11PM (#27438763)
It’s hanging out in a private repo with a bunch of other stuff and I don’t care to put it in it’s own repo at the moment. Basically CloudFormation publishes a JSON spec of all of their resource types and I use that to generate Python code with type annotations. It’s sort of like Troposphere, but I go further—Tropo makes you reference resources by their cloudformation string names, but my tool lets you use the Python object containing the resource and it will resolve to the correct CloudFormation “Ref” object at compile time. (also, unlike tropo, I generated my Python types from a spec so I don’t have to keep up with AWS changes). That said, I’ve given up on CloudFormation altogether since Terraform has better support for resources outside of AWS.
Re:Terraform 1.0 by nuker on Tuesday June 08, 2021 @10:03AM (#27435340)
>if you want to create a new resource, you have to run it as its own lambda

Please don't, lol

Re:Terraform 1.0 by kubanczyk on Tuesday June 08, 2021 @11:17AM (#27436316)
> they also have a CDK

Terraform-CDK, as of now, needs to go through standard HCL parser. Sadly, there is no backdoor into Terraform's internal structures. If HCL (as a language) is the limitation for you, the CDK does not let you fly around it.

Re:Terraform 1.0 by smaddox on Tuesday June 08, 2021 @10:59AM (#27436058)
This would be great. Perhaps it could be based on https://dhall-lang.org/
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @11:13AM (#27436259)
I absolutely think a statically typed language is the right way to go (from experience using a Python->CloudFormation generator even with Mypy), but Dhall is going to be really unfamiliar for most people and it's hard to sell people on new languages that are syntactically unfamiliar.

As an aside, I think functional concepts could have made their way into mainstream programming much earlier if the FP people would have been willing to lower themselves to syntax that is readable to us plebs--I think this is no small part of Rust's success. People say syntax doesn't matter, but I disagree.

Re:Terraform 1.0 by verdverm on Tuesday June 08, 2021 @12:45PM (#27437575)
https://cuelang.org has better syntax but its logic based unification is a struggle bus for many people.
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @04:09PM (#27440109)
I looked at Cue and I don't understand what problem it solves. It certainly doesn't (seem) to solve the problem of DRYing up verbose YAML, or at least it's missing any notion of a function.

"hey, these YAML blobs are all mostly the same, but they vary based on a couple of parameters--I should write a function that takes those parameters and outputs the right YAML object"

^ This is the #1 thing that the high-level language should concern itself with. Static typing is really nice to have and it's cool that Cue has a pretty interesting type system, but (as far as I can tell) it doesn't have functions. It almost has functions, but I don't want to have to resort to a hack for the #1 thing that I care about (functions).

Considering I prefer functions over sane syntax (although sane syntax is roughly tied with static typing), I'm inclined to prefer Dhall over Cue, but I'm still optimistic that something better will emerge. Also while we're on syntaxes that are deliberately obtuse, I'm pretty sure the Nix community has a Nickel language which is basically a statically typed version of the Nix language.

Maybe Cue has a more enlightened way of thinking about the infra-as-code problem and I'm just not getting it.

Re:Terraform 1.0 by verdverm on Tuesday June 08, 2021 @04:40PM (#27440474)
CUE's philosophy is to wrap code in data, not data in code, as learned from the major configuration systems at Google. Being a logical language, rather than telling the computer what to do, you state facts and it verifies that you are correct. It is also intentionally not Turing complete do that you cannot program in CUE.

CUE is gaining traction while still being young and changing. Grafana is adopting it for validating dashboards and such. Expect to see it more in DevOps too

Re:Terraform 1.0 by randomswede on Thursday June 10, 2021 @03:55AM (#27458113)
When I stopped being an SRE at Google, my most immediate thought was relief that I would never, ever, have to deal with BCL/GCL again.

After 6 months outside Google, I desperately wished for BCL/GCL to be everywhere, because all other config languages were just plain broken. And more annoyingly, there's no better way to describe it than "I have seen better, just trust me".

CUE seems to be a step forward. Flabbergast looked like it might have been a contender. The latter is DEFINITELY inspired by BCL/GCL.

At some point, I will have to sit down with CUE and try to re-implement the "perfect little horror" in it (it should be impossible IFF CUE is not Turing-complete, but it actually turns out that there are edge cases of configuration where you want that Turing-completeness).

Re:Terraform 1.0 by throwaway894345 on Wednesday June 09, 2021 @02:29PM (#27451355)
> Being a logical language, rather than telling the computer what to do, you state facts and it verifies that you are correct.

Sure--it's like advanced static typing for static configuration. But that seems like a different and lesser problem than DRYing up the configuration in the first place, and moreover if you use a statically typed programming language to DRY up your configuration then you get pretty similar guarantees to Cue. You don't get Cue's "unifying many definitions" approach, but I can't honestly discern the value proposition in that.

As for turing incompleteness, that's a nice to have at best. If I had to choose between a turing incomplete declarative language like JSON and a turing complete imperative language like Lua, I'd take the latter every single time.

Re:Terraform 1.0 by lmm on Tuesday June 08, 2021 @10:13PM (#27443481)
Nah, the syntax is superficial. Scala has offered better-than-Rust FP in a traditional syntax for over a decade, but if anything the tension between imperative and functional people is worse there.
Re:Terraform 1.0 by garethrowlands on Tuesday June 08, 2021 @12:58PM (#27437747)
Re:Terraform 1.0 by kortex on Tuesday June 08, 2021 @11:00AM (#27436075)
You can reduce a little bit of the repetition in YAML with anchors.

There are tools that convert JSON/YAML into HCL.

https://learnxinyminutes.com/docs/yaml/#:~:text=yaml%20also%...

Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @11:08AM (#27436194)
I think you misunderstand the problem I'm trying to solve, or maybe I misunderstand your response. My goal isn't to write YAML instead of HCL, my goal is to get rid of HCL and Terraform semantics altogether. If I had my way, Terraform's low level engine would operate on a verbose (i.e., "not DRY") YAML (or JSON or HCL or I don't care) description of resources which would be generated from (for example) a Python script.

The Python/Go/etc script is what humans interface with, and it is DRY. The YAML/HCL/etc is what the Terraform engine operates on and humans should very rarely need to interact with this.

Re:Terraform 1.0 by kortex on Tuesday June 08, 2021 @12:34PM (#27437426)
Ah, so like you have some process which generates your YAML/HCL, which is your "IR/assembly" layer, not meant for regular human consumption/editing, which is fed to Terraform. But it's readable/auditable, VCS-trackable, and diff-able.

I do that a lot as well and in fact I'm kinda leaning towards taking that approach from the get-go. Right now I start with the YAML, but then something makes inevitably leads me to templating it using make + jinja/gomplate, which eventually leads me to wanting to use python scripts, and then invoke (python package, it's like gulp or make).

It's not code, like business logic code, but it's too verbose and repetitive for human manual editing.

Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @04:04PM (#27440058)
Yeah, in the Kubernetes world, the official interface is the YAML/assembler and different people have come up with different approaches for generating that. Helm for a long time (and even currently) uses text templates (e.g., jinja, mustache, etc) to render that YAML which is predictably abysmal.

CloudFormation used JSON (and eventually YAML) but built on top of it language-like facilities (the ability to reference resources, call pseudo-functions, etc) all very poorly. So you get an impoverished language built on top of YAML.

Terraform decided they would do approximately the same thing, except they reinvented their own JSON/YAML alternative (HCL) and built a crappy programming language atop it (instead of atop JSON/YAML).

These all give you pretty crumby means of abstraction. CloudFormation you get nested stacks instead of functions and you can only pass scalars around (no objects or lists--except comma-delineated strings which can be parsed into a list of strings). You're also limited in how many nested stacks you can create and how many total parameters can be passed into any given top-level stack.

Terraform seems strictly better. You can pass objects and lists and I've never approached any parameter limits, but still, you have to create a whole directory just to define a function and refactoring existing code into a module is painful because it means renaming resources (putting them under the module) which Terraform interprets as intent to destroy and recreate the resource.

Helm is using text templates so you can even generate syntactically invalid YAML! I think they might be supporting Lua these days, but I haven't looked into it.

I think the idea was that the whole marketing push behind infra as code was "it's just YAML! Such declarative! Wow!" as though yaml magically simplifies the inherently complex task of infrastructure, so everyone started with something YAML-like--even though we absolutely should have known that we would need to abstract--and gradually built our own half-baked languages on top of them. Of course, infra as code is absolutely worthwhile, but it's the ability to define what you want and have a tool reconcile it with some current state--it's not some magical property of YAML/JSON/HCL/etc.

fin.

Re:Terraform 1.0 by zbentley on Wednesday June 09, 2021 @01:20AM (#27444473)
That's an accurate summary of the arc of progress in this area. Also explains why so many folks are now turning to operators (versioned procedural code that runs in k8s and does arbitrary things, rather than arbitrary versioned yaml artifacts applied to k8s) to do advanced stuff rather than layering on more templating duct tape.
Re:Terraform 1.0 by appleflaxen on Tuesday June 08, 2021 @09:34AM (#27434950)
> these tools should have a static/yaml-like "assembly language" that describes the state of your infrastructure without any of the DRY

the last five words are a bit of a double negative; i think you mean "without the repetition" but I can't tell.

Re:Terraform 1.0 by Hasnep on Tuesday June 08, 2021 @09:46AM (#27435143)
"without DRY" in this case means "with repetitions" i.e. in a verbose way. GP wants to be able to generate this verbose, machine readable syntax with DRY, human readable syntax.
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @10:22AM (#27435565)
Yes, this. Thanks for clarifying for me, apologies to the parent for my lack of clarity.
Re:Terraform 1.0 by MadVikingGod on Tuesday June 08, 2021 @09:54AM (#27435239)
Dang, your solution sound so much like kubernetes I'm not sure if you are joking or not.
Re:Terraform 1.0 by throwaway894345 on Tuesday June 08, 2021 @10:27AM (#27435626)
Kubernetes is one conceivable incarnation, but it operates differently than other infra-as-code tools. Terraform, for example, builds a dependency graph of your resources and initializes them in order. Kubernetes doesn't care about dependencies, and it just keeps trying to create resources and things will fail until their dependencies come online.

Further, Kubernetes manifests are the verbose "assembly language" layer, so you still need something for humans that is DRYer.

We use Terraform to manage Kubernetes resources (as well as cloud provider resources) at the moment, but I think you can equally use cloud provider operators for Kubernetes and manage everything with Kubernetes--I haven't tried this yet so I can't comment. In the latter case, you would still need something to DRY up your Kubernetes manifests. Also, if you aren't running on Kubernetes and you just want infra-as-code, k8s is an expensive solution (in terms of operations).

What I was picturing was a more conventional infra-as-code diffing engine (like Terraform's) but with a more verbose interface similar to Kubernetes YAML.

Re:Terraform 1.0 by zbentley on Wednesday June 09, 2021 @01:24AM (#27444498)
> Kubernetes manifests are the verbose "assembly language" layer, so you still need something for humans that is DRYer.

It's a little more than that. Out-of-the-box manifests for primitives are certainly assembly-like, you're right--but CRDs allow you to operate at a higher level of abstraction while staying in the same syntax, which is powerful and unique to k8s (everything else, from Helm to Terraform to Ansible, distinguishes between pseudo-assembly "language that directly expresses changes to be made" and "language that humans can write abstractions in").