+1 for the question about a "standard" or recommended way to parse and generate .tf files from a script.

I have two use cases in mind: Use case 1: need to have a reproducible way to generate terraform folders for multiple almost-identical deployments (this can be solved via modules) Use case 2: need way to "promote" a deployment from staging to production (load a .tf file, change a few basic params then save again to a different folder --- doesn't really work with templates since I want to load an existing .tf file)

One thing that might work is to use the combination of two tools: - https://github.com/virtuald/pyhcl = reads .tf, can export .tf.json - https://github.com/kvz/json2hcl = convert .tf.json to .tf but feels hacky...

Any other recommendations for .tf parsing and generation? (preferably in python or scriptable via python)

Your favorite tool that produces json should be able to work. I'd probably use jsonnet, which has a multi-file mode so that one template file can render into many json files.

You can actually use JSON files in place of HCL: https://www.terraform.io/docs/configuration/syntax-json.html

main.tf.json

You can't write a local module for the resource that takes the differences as inputs?

Maybe I can, but I am failing to grasp how templates or the modules work, we have these resources where only the ami is different (and sometimes the count):

    resource "aws_instance" "nomadclients_tick" {
      ami = "${data.aws_ami.nomadclient_tick.image_id}"
      instance_type = "t2.micro"

      count = 3

      iam_instance_profile   = "${aws_iam_instance_profile.consul-join.name}"
      subnet_id              = "${element(aws_subnet.consul.*.id, count.index)}"

      vpc_security_group_ids = [
        "${aws_security_group.rule01.id}",
        "${aws_security_group.rule02.id}",
        "${aws_security_group.rule03.id}",
        "${aws_security_group.rule04.id}",
      ]
    }

    resource "aws_instance" "nomadclients_tock" {
      ami = "${data.aws_ami.nomadclient_tock.image_id}"
      instance_type = "t2.micro"

      count = 3

      iam_instance_profile   = "${aws_iam_instance_profile.consul-join.name}"
      subnet_id              = "${element(aws_subnet.consul.*.id, count.index)}"

      vpc_security_group_ids = [
        "${aws_security_group.rule01.id}",
        "${aws_security_group.rule02.id}",
        "${aws_security_group.rule03.id}",
        "${aws_security_group.rule04.id}",
      ]
    }

The examples I have found feels very difficult.

I personally would solve this issue with a local module (very easy to set up, just a subfolder with a main.tf that defines the resources[0] then it's called as here[1]) with two input variables[2]: a required ami variable, and an optional count variable which has a default of 3. I hope that helps.

[0] https://www.terraform.io/docs/modules/index.html

[1] https://www.terraform.io/docs/modules/sources.html#local-pat...

[2] https://www.terraform.io/docs/configuration/variables.html

One way I can think of that will solve this problem is to do something like:

    resource "aws_instance" "nomadclients" {
      ami = "${count.index > 2 ? data.aws_ami.nomadclient_tick.image_id : data.aws_ami.nomadclient_tock.image_id}"
      instance_type = "t2.micro"

      count = 6

      iam_instance_profile   = "${aws_iam_instance_profile.consul-join.name}"
      subnet_id              = "${element(aws_subnet.consul.*.id, count.index)}"

      vpc_security_group_ids = [
        "${aws_security_group.rule01.id}",
        "${aws_security_group.rule02.id}",
        "${aws_security_group.rule03.id}",
        "${aws_security_group.rule04.id}",
      ]
    }

The downside to this approach (in TF < 0.12), however, will become apparent when you want to modify the number of instances in one or both pools. This arrises from the way hcl v1 manages the counter index with each resource in the state; That is each resource is linked to a specific index and when that index is altered in someway, terraform will attempt to adjust the resources at each respective index accordingly.

Because of these headaches, we decided to abandon hcl for these use cases and ended up writing our own preprocessor that takes a JSON configuration and generates individual JSON-based HCL (ie. .json.tf) that terraform can then use. We can leverage proper templating to generate many variants of a single resource in a higher level language, while still using terraform to manage the infrastructure and it only adds one additional step to the plan + apply process.

Alternatively you could put the AMIs in an array, and just iterate over that array.

  locals {
    nomad_ami = [
      "${data.aws_ami.nomadclient_tick.image_id}",
      "${data.aws_ami.nomadclient_tock.image_id}",
    ]
  }

  resource "aws_instance" "nomadclients" {
    count = 6
    ami   = "${element(local.nomad_ami, count.index)}"
  
    ...
  }

How does a var, template, or even a generic resource that runs a local script not solve this problem?