You’re not telling AWS how to do something you’re telling it what you want the end state to be and let it figure out what needs to be created, updated, deleted, or replaced and the dependency chain what can be run in parallel when you create or update the template.
There are linters and editors for CloudFormation that help you with autocomplete (?) and to warn you when you are specifying a a resource type. You can even add your own custom definitions to the linters for custom resource types that you create.
CloudFormation just like generic SQL doesn’t by itself have the concept of loops. CloudFormation does support custom transforms and macros that you can create in any language and you can write programs that generate CF in many languages using the CDK where it will perform validation. I haven’t used it so I am being really hand wavy.
We ended up preprocessing them using jinja2, injecting our variables via its context, and using its cleaner syntax for expressing conditions, loops, etc. Now we have the best of both worlds.
Apologies, I don't mean to be a downer. But... you're making your present life easier but making the code harder to maintain for future maintainers of your codebase.
I’m eagerly awaiting 0.12 so the language would have similar features as jinja templating.
That's what sparkleformation does for cloudformation stacks.
Just as in databases, you end up playing lots of “read the query plan, restate the query to try and get a better one” games.
For all the many benefits we've seen with Infrastructure as Code to date, the tools are still fairly primitive - copy/paste is the norm for reuse, testing is rare or non-existant, productivity during infrastructure development is low, continuous integration and delivery are largely ad-hoc, and there are very few higher-level libraries available to abstract away the complex details of today's modern cloud platforms. Net - it feels like we're still programming cloud infrastructure at the assembly-language level.
I'm really excited about the opportunity to bring more software engineering rigor into the infrastructure as code space. At Pulumi we believe using existing programming languages is a key enabler of this. Pulumi is still a desired state model like other Infrastructure as Code offerings (so you can still preview changes and make minimal deltas to existing infrastructure) - but you can write code to construct that desired state. As a result, you get for loops and conditionals, you get types and error checking, you get IDE productivity tooling, you can create abstractions and interfaces around components, you can write tests, you can confidently refactor you infrastructure code, you can deliver and version components via robust package managers, and you can integrate naturally into CI/CD workflows.
Pulumi isn't the only tool in this space - we're seeing things like Atomist bringing this same model to delivery pipelines, and AWS CDK bringing this model to the CloudFormation space. I'm excited about where these tools will take the Infrastructure as Code ecosystem in the coming years.
[disclaimer - CTO at https://pulumi.io so clearly biased on this topic :-)]
So good luck to you!
I saw it with early versions of Chef, I've seen it with Apache Auroras python based job specs for Mesos etc.
I'd rather work around the limitations of a DSL that is declarative and limited but is consistent across orgs than having to retrain myself if I move to a new job.
Operations is the wrong place to do engineering, either that or operations needs to level up significantly since they do cause a lot of delays and damage to companies with these poorly design tools.
Configs are hard, and they're different to enough from normal software (for example DRY doesn't apply in the same circumstances) that using the same tools is a bad idea.
I've seen what happens when swes use swe languages on configs. They get unintelligible. And then I have to clean up the messes.
To be clear though, using Json or toml for configs is also often a mistake.
I think a valid question would be what is the right framework factoring or design should be, but pure declarative isn’t not and half declarative like Terraform are just repeating past mistakes.
Doesn't it, though? Dumb configs tend to suffer very much because of the impossibility to apply DRY directly, and people end up using/writing config generators just to ensure consistency of values.
Hell, isn't half of the job of a IaC tool to be such a config generator, papering over lack of capabilities of configuration languages?
Theoretically, maybe. In practice, I don't think so. Mostly because code is much less prone to change than configs are. Like if you have some encapsulated set of behavior in code, it's often easy (and not particularly painful) to do something like
self.thing = x if self.other else default
This is (normally) fine. Sometimes this will expand to be 2-3 different things. So you might get if self.other:
self.val = ...
elif self.different_situation:
self.val = ...
else:
self.val = default
And this avoids setting self.val in all of the child classes, it's "nice and DRY".Because code-code doesn't change that much, this kind of weird gross encapsulated behavior is (usually) ok. It's still a code-smell, but you don't get burned by it. But when you are dealing with configs, they do change, and you want to be extra explicit, because (as you probably know), the number of exceptions and special cases will inevitably grow, and you'll end up with implicit leaf-level configuration hidden away in your so-called encapsulated stuff, but without end-user visibility into what is actually happening.
One solution to this is to completely ignore DRY and ensure consistency via unit tests or static analyzers or something, but that also sucks (possibly more) than doing some denormalization within your config-language itself.
My experience says the right way to do this is to restrict yourself to not using any conditionals other than perhaps get-with-default, and doing everything that would be done conditionally via inheritance or composition. Remains to be seen if I think I made the right decision 5 years from now.
I find this a strange argument. Traditional operations people will still resist using these techniques, and will use pre-canned modules and resources if they're forced to use something.
The real reason seems to be because of the declarative nature of IaC rather than concerns about adoption.
(Those examples, the real world ones, where from competent software developers that did product development, not operations people).
If you’re in technology your profession is to solve problems, it’s not to entrench poor decisions in a company, which in some sense everyone is at fault for, but this area affects a lot of a company and the attitudes are pretty anti-improvement if it goes out of someone’s skill set. The end result is that the software will be moved out of operations, I see this more and more. Operations can’t improve just by magic, they need to embrace the past 50 years of computing knowledge too.
Oh. Really? /s
That's literally what DevOps is all about. I'm sorry but Infrastructure isn't as simple to manage in code as "regular" software. The Infrastructure API's mutate or behave in unexpected ways too often. State drift is common. There is active work in figuring out better solutions. But to say that operations have not embraced computing knowledge is nonsense.
Pulumi is going down a better route in that they’re using a normal programing language with a normal tool set, however I’m skeptical of how their engine is designed with RPC calls for language bindings, since again it makes debugging more complex as opposed to just a normal sdk. They also don’t have debugging enabled yet.
What I see as optimal is tool like make, which calls other command line tools. Resolves dependencies, processes errors. But not make, something with sane syntax and error handling.
All these new fancy all-Go tools creep me out. Infrastructure tooling should not live in domain of one programming language. Extensibility should be language agnostic. If I want to write some Perl/Python/Bash script to support some very non standard part of my infrastructure, I should be able to. If I want to plug vendor specific utility execution into deployment pipeline, I should be able to. And it should be easy.
It certainly makes one's ability to look into the machinery and know what is going on much harder.
> If I want to write some Perl/Python/Bash script to support some very non standard part of my infrastructure, I should be able to
That very feature is why I love ansible: if there is some quirk, or even an unreleased module (they only ship new features in major releases, I recently learned), then you can copy the upstream file, or a modified version of the existing one, or even a whole new module, into the `library` or `lookup_plugins` directory of your playbook and you're back in business. No fighting with golang anything. You can also write ansible modules in any language you like.
The counterpoint to this is things like Gulp or Gradle which become a nightmare after a couple years of multiple developers and coding styles appending things here and there. Now rather than just spending a few hours learning a basic config DSL, I have to build up a mental execution model every time I want to add a build step.
Similar to React and it's virtual dom.
ShellJs works pretty nice. Not only is my script now cross platform, but doing conditional logic and user prompting is a lot easier in code than bash.
The only "issue" I've found is that ShellJs is quite barebones. I wrote a wrapper over it to do everything, such as nice question prompting and colored output.
I used to use this heavily back in 2013, 2014. Infochimps got picked up by eBay, afair. Hence why this was never developed further.
I disagree with this one. I believe describing an infrastructure should be fully declarative and the tool should decide how it needs to create resources based on the description. This way, the infrastructure code almost can't have bugs, but the tool can be fixed for everybody.
Imho tools should use actual code (whether it's TypeScript or Kotlin or whatever) instead of reinventing constructs like loops and string interpolation.
Thankfully these tools are getting more popular, because frankly I can't stand configuring another Kubernetes or GCP resource using a huge block of copy/pasted YAML.