You can feed JSON to Terraform however this falls over if you need dependencies for output values. This usually isn't an issue because most Cloud provider resources have predictable IDs but as soon as you have one that doesn't you are up for a lot of pain and suffering.
Basically it's Terraform but instead of declaring your resources in HCL, you declare them in a real programming language. You're still producing a declarative config that the engine then diffs, applies etc. In fact, it's compatible with existing terraform providers, so it has a surprisingly large selection of things you can use it for.
Note their docs will try to guide you towards using their hosted service which basically does nothing except host the state file, but you can use an S3 or GCS bucket instead and it works fine.
It's definitely not without its own problems, but I'd say it's overall an improvement.
{
"resources": [
{
"type": "aws_s3_bucket",
"properties": {"name": "foo"}
}
]
}
Note that there is no reference to "foo_bucket".A resource provider defines a collection of fields that is the "identifier" for the resource. For example, an S3 bucket resource would have the "name" field for its identifier.
If you change another attribute besides the bucket name, the engine will see that the input and the state both have a s3 bucket resource with the same name but different props, so it knows it will need to update some props (rather than create a new one). However, if the name changes, the engine will see that the input has a bucket that doesn't exist in the state so it will add a "create bucket" step to the plan. It will also see that the state has a bucket that isn't in the input, so it will add a "delete bucket" step to the plan.
Maybe another way of saying the same thing is that a resource provider can mark any given field as "forces replacement", and all of the fields that force replacement are the de facto identifiers? I haven't thought through whether these are exactly equivalent.
And back to the s3 bucket use case sometimes you want uniqueness in your name so you use a prefix instead of specifying the whole name - how do you determine which bucket that resources is referencing if there are multiple buckets matching the prefix?
I hear what you're saying in terms of wanting state management to be simplified, but pretty much every IaC solution uses this explicit logical resource -> physical resource mapping in state.
Edit: the aliases would have to handle moving as well as renaming. You could just have aliases in a global namespace, which means adding `alias = "portable-elb"` and doing one `terraform apply` means you can pick up that config, drop it anywhere else, and it will move it for you. It wouldn't even need to do a full `apply`, just a local JSON manipulation.
I get it from HashiCorp's perspective though.
A robust application engine with a suboptimal config generator is a viable product.
A suboptimal application engine with a brilliant config generator is not.
So given limited resources, former gets the dev grease.
You can generate these configs really easily with any off-the-shelf programming language for a small fraction of the effort they’ve put into HCL + all of the stuff on top that makes HCL the shitty programming language that it is.
Even if you insist on building your own programming language for this purpose, Hashicorp could’ve saved themselves a lot of work by looking at the prior art of the last 70 years of programming language history.
In other words, if they just picked, say, JavaScript from the start they could have saved a bunch of time and energy and put that into their application engine.
This is what I've started doing with Jsonnet for generation, and also exactly why I've stopped doing it.
CloudFormation ?
> There would be a diffing engine which would figure out what changes need to be applied and apply them accordingly.
CloudFormation.
Tools are never 'just tools', there is context and there are externalities. And as you already pointed out: migrating/uprooting all of those other things isn't a likely scenario.
care to share? (I know some hn users often don't w/o being asked, out of a sense of not wanting to be seen as self-promoting.)
Please don't, lol
Terraform-CDK, as of now, needs to go through standard HCL parser. Sadly, there is no backdoor into Terraform's internal structures. If HCL (as a language) is the limitation for you, the CDK does not let you fly around it.
As an aside, I think functional concepts could have made their way into mainstream programming much earlier if the FP people would have been willing to lower themselves to syntax that is readable to us plebs--I think this is no small part of Rust's success. People say syntax doesn't matter, but I disagree.
"hey, these YAML blobs are all mostly the same, but they vary based on a couple of parameters--I should write a function that takes those parameters and outputs the right YAML object"
^ This is the #1 thing that the high-level language should concern itself with. Static typing is really nice to have and it's cool that Cue has a pretty interesting type system, but (as far as I can tell) it doesn't have functions. It almost has functions, but I don't want to have to resort to a hack for the #1 thing that I care about (functions).
Considering I prefer functions over sane syntax (although sane syntax is roughly tied with static typing), I'm inclined to prefer Dhall over Cue, but I'm still optimistic that something better will emerge. Also while we're on syntaxes that are deliberately obtuse, I'm pretty sure the Nix community has a Nickel language which is basically a statically typed version of the Nix language.
Maybe Cue has a more enlightened way of thinking about the infra-as-code problem and I'm just not getting it.
CUE is gaining traction while still being young and changing. Grafana is adopting it for validating dashboards and such. Expect to see it more in DevOps too
After 6 months outside Google, I desperately wished for BCL/GCL to be everywhere, because all other config languages were just plain broken. And more annoyingly, there's no better way to describe it than "I have seen better, just trust me".
CUE seems to be a step forward. Flabbergast looked like it might have been a contender. The latter is DEFINITELY inspired by BCL/GCL.
At some point, I will have to sit down with CUE and try to re-implement the "perfect little horror" in it (it should be impossible IFF CUE is not Turing-complete, but it actually turns out that there are edge cases of configuration where you want that Turing-completeness).
Sure--it's like advanced static typing for static configuration. But that seems like a different and lesser problem than DRYing up the configuration in the first place, and moreover if you use a statically typed programming language to DRY up your configuration then you get pretty similar guarantees to Cue. You don't get Cue's "unifying many definitions" approach, but I can't honestly discern the value proposition in that.
As for turing incompleteness, that's a nice to have at best. If I had to choose between a turing incomplete declarative language like JSON and a turing complete imperative language like Lua, I'd take the latter every single time.
There are tools that convert JSON/YAML into HCL.
https://learnxinyminutes.com/docs/yaml/#:~:text=yaml%20also%...
The Python/Go/etc script is what humans interface with, and it is DRY. The YAML/HCL/etc is what the Terraform engine operates on and humans should very rarely need to interact with this.
I do that a lot as well and in fact I'm kinda leaning towards taking that approach from the get-go. Right now I start with the YAML, but then something makes inevitably leads me to templating it using make + jinja/gomplate, which eventually leads me to wanting to use python scripts, and then invoke (python package, it's like gulp or make).
It's not code, like business logic code, but it's too verbose and repetitive for human manual editing.
CloudFormation used JSON (and eventually YAML) but built on top of it language-like facilities (the ability to reference resources, call pseudo-functions, etc) all very poorly. So you get an impoverished language built on top of YAML.
Terraform decided they would do approximately the same thing, except they reinvented their own JSON/YAML alternative (HCL) and built a crappy programming language atop it (instead of atop JSON/YAML).
These all give you pretty crumby means of abstraction. CloudFormation you get nested stacks instead of functions and you can only pass scalars around (no objects or lists--except comma-delineated strings which can be parsed into a list of strings). You're also limited in how many nested stacks you can create and how many total parameters can be passed into any given top-level stack.
Terraform seems strictly better. You can pass objects and lists and I've never approached any parameter limits, but still, you have to create a whole directory just to define a function and refactoring existing code into a module is painful because it means renaming resources (putting them under the module) which Terraform interprets as intent to destroy and recreate the resource.
Helm is using text templates so you can even generate syntactically invalid YAML! I think they might be supporting Lua these days, but I haven't looked into it.
I think the idea was that the whole marketing push behind infra as code was "it's just YAML! Such declarative! Wow!" as though yaml magically simplifies the inherently complex task of infrastructure, so everyone started with something YAML-like--even though we absolutely should have known that we would need to abstract--and gradually built our own half-baked languages on top of them. Of course, infra as code is absolutely worthwhile, but it's the ability to define what you want and have a tool reconcile it with some current state--it's not some magical property of YAML/JSON/HCL/etc.
fin.
the last five words are a bit of a double negative; i think you mean "without the repetition" but I can't tell.
Further, Kubernetes manifests are the verbose "assembly language" layer, so you still need something for humans that is DRYer.
We use Terraform to manage Kubernetes resources (as well as cloud provider resources) at the moment, but I think you can equally use cloud provider operators for Kubernetes and manage everything with Kubernetes--I haven't tried this yet so I can't comment. In the latter case, you would still need something to DRY up your Kubernetes manifests. Also, if you aren't running on Kubernetes and you just want infra-as-code, k8s is an expensive solution (in terms of operations).
What I was picturing was a more conventional infra-as-code diffing engine (like Terraform's) but with a more verbose interface similar to Kubernetes YAML.
It's a little more than that. Out-of-the-box manifests for primitives are certainly assembly-like, you're right--but CRDs allow you to operate at a higher level of abstraction while staying in the same syntax, which is powerful and unique to k8s (everything else, from Helm to Terraform to Ansible, distinguishes between pseudo-assembly "language that directly expresses changes to be made" and "language that humans can write abstractions in").
I know they also have a CDK, but I can't tell if it properly solves that problem or if it still forces us into Terraform idiosyncrasies (i.e., if I rename something in Terraform, it will try to delete the corresponding resource and recreate it, and I think that absurd behavior remains with the CDK).