Here in lie much of the ”devops” problem, imo, as coders seem to want to allow easy configuration management through “simple“ declarations, inventing almost a new language in the process.
Having worked for 20 years with systems/configuration management, mainly as a developer I can tell many coders have not. Not at scale, and not supporting 100s of different services simultaneously. Hence square wheels gets re-invented.
To be rid of all yaml templating and loosely coupled technical integrations you will have to treat config/service delivery/ci/cd/what-have-you as a business domain of it’s own, and develop it the same way.
I guess we agree, completely perhaps?! :)
I’ve been fortunate to have been able to work everything from small upstart to big corporate enterprises and to me this is more true now than ever.
As a developer I build my own version of a CMDB, I do not use crazy templating - I write an API, that leverages the CMDB and produces sane output. It’s still configuration as code, just something that is easier to scale and adapt.
Guess I’m only disagreeing on whom to blame for this mess.
If we step back, there are a few different strategies for dealing with operational logic. From least scalable to most (roughly):
1. Human processes (ex: checklist)
2. In the config itself (ex: Kubernetes' YAML)
3. A config generator (ex: Helm, Kustomize)
4. Software (ex: Kubernetes Operator)
I see a lot of people get stuck doing one for too long.
When doing one becomes unwieldy, it's time to consider the next. That or find a way to reduce the need for all that operational logic.
The point of all this is that software configuration is never a simple declaration. It is always a mess of behaviors and derived values.
I am not sure I totally agree, but it at least correct enough to always break something. And adding abstraction layers just because you choose to use an underpowered notation at the bottom isn't a good practice.
(It's the same with SQL, you describe what you want back rather than how its achieved. A declarative approach works fairly well there.)
Thus, it is very easy to get in a situation where an Ansible playbook applied against two machines with slightly different production history will result in very different behaviour.
If you want real declarative configuration management, try Nix/NixOS.
Terraform on the other hand does have a concept of ownership, diff-ing, and applying changes. It takes some work as you now need to track state, but I've been very happy with Terraform.
Sure, that's true with many of these, so what?
The problem isn't that the languages are declarative—functional code is declarative.
The problem is that they are extremely limited in their expressive capabilities, so that it is much more complex and error prone to describe the final state in them than it would be—even in a declarative style specifying the final configuration—in a more complete language than YAML (or, in many cases, a language essentially limited to JSON’s expressive capability even if it also supports a YAML serialization.)
> It's the same with SQL, you describe what you want back rather than how its achieved.
YAML would be an inadequate alternative for SQL’s role, too.
It's a hard to tell where marelle ends and the config begins, since it's all just prolog.
What would be the benefit of adding a declaritive language into the mix over YAML?
Emacs is configured using Lisp. Because of that it is amazingly configurable. XMonad uses Haskell, which gives types to avoid lots of error cases.
The thing is providing an actually programming language doesn't remove anything specially if the same config can be written as cleanly.
And you would avoid "YAML templates for creating YAML", when you actually need to process some data (even if it's just for pre/suffix creation in names). Secrets retrieval is another thing that you also need to do and template.
Also, YAML is a terrible to parse language, with can give out weird error cases. Other languages compilers/interpreters are more mature.
And ofc, if you provide SDKs for your IaC tool, you should be able to use the language that your developers are more familiar with. Taking advantage of the good practices their are used to. Don't limit with "declarative languages". Use a programing language that make more sense, and leave data languages for data.
YAML is not typed and you can basically do whatever. The worst of all worlds.
The capability to readily define, store in libraries, and use reusable abstractions that apply within a configuration or across multiple individual configurations.
Of course, you could write code to generate your YAML, but the tooling is _not_ going to help you with that today.
Maven and Gradle are examples of each of these methods -- both build up an internal model of the build, but while Maven uses XML, Gradle uses a Groozy DSL.
And LUA has its own hype, just not in configuration scenarios.
But at least there is the AWS CDK and Pulumi now to enable “infrastructure as code” as code.
As a big proponent of Dhall, I gotta say, probably the primary reason for that at this point is due to a lack of language bindings. Most tools in the Kubernetes ecosystem are written in Go. Currently, there is no official Go language binding for Dhall - one implementation[1] is pretty close, but the marshalling features need some work, and it's not the author's day job. The only way to get Dhall configuration into a Go program today is to either shell out and call the Haskell-based reference implementation directly or to script your Go binary such that the Haskell-based reference implementation translates the Dhall configuration to JSON and then feeds that into the Go program. Even that's not ideal - you end up maintaining your schema in Go code and in Dhall simultaneously.
I very much believe that Dhall will solve the stringly-typed insanity which you refer to, but the language is not quite there yet. While I'd love for some of the Kubernetes core developers to step up, I mean, I can understand people's unwillingness to adopt something that isn't 100% perfectly well-supported and handed to them on a silver platter.
Using Dhall to create the YML you will use isn't great, but it's quite possible today and is much better than using YAML to script a YAML generator that will read some more YAML to create the YAML you will use.
I guess some people simply like YAML.
DSC or pure powershell has bunch of useful tools and a real programming language to use in between. Yaml like configuration can be achieved with nested HashTables that have almost equivalent readability without loosing any of the devlopment capabilities.
Sometimes one doesn't "need" a coder because the system is such that it wouldn't do you any good anyway.
IMHO a big part of the problem is that the companies involved are actively incentivized to justify their existence by making things more complicated instead of less complicated. If things are too simple, nobody needs their consulting, training, support, etc. Worse most of these companies are competing for attention and typically only provide part of the solution instead of the whole solution. So there's a lot of money in cross integrating multiple complicated solutions from different companies. Indeed many of the companies involved are serial offenders when it comes to this. As soon as you get companies like Red Hat, Amazon, Oracle, etc. get interested in stuff, brace yourself for some convoluted vendor lockin. That's basically what happened to Kubernetes.
When it comes to languages, I'd love something that has a compiler and auto complete. I'd also like something that has a notion of internal domain specific languages. IMHO Typescript, Kotlin or Rust could work for this. The point of this would be leveraging a type system for static checks and code completion. That kind of rules out dynamic languages like javascript, python, or ruby (all of which have been used in devops in the past). They are nice but not nice enough.
However, typescript has enough static typing that you get some meaningful autocomplete. Also, it's already popular in the node.js world (i.e. lots of stuff available to integrate). Kotlin would also be able to fit in that world since there's a transpiler to javascript, it can utilize typescript type definitions and it integrates with node.js and npm. Rust would be a bit of an outlier here and probably a bit too much in terms of language complexity for the average yaml loving devops type with a mere few years of full stack experience. With Kotlin, the transition of gradle from groovy to Kotlin has been interesting. Groovy basically is a dynamically typed language and they've slowly been replacing it with the more strictly type Kotlin. That sort of stabilized with v5 and with v6 it got usable for a lot of projects and recent updates to Intellij have improved how it is able to support developers with e.g. code completion and other features. Something like that aimed at devops could work.
General purpose programing languages are getting more expressive by the day, why do we use data serialization languages instead for configs? it doesn't make any sense.
Configuration is code not data.
You know, once upon a time, we understood that declarative approaches to software engineering were superior to imperative approaches, when declarative approaches are feasible. Declarative approaches are much safer and easier to test, at a cost of only being able to express what the tool accepting the declarative approach can understand. Imperative approaches are strictly worse for any problem set where a declarative approach solves the problem within performance requirements. The additional expressiveness of languages like Pulumi is the last thing I want.
YAML is a horrible language for declarative system configuration because a) any sufficiently complex system will require you to generate your declarative codebase in the name of maintainability, b) generating code for any language where whitespace is significant will lead you to an early death, and c) stringly-typed languages are fundamentally unmaintainable at sufficient scale. But this is not an indictment of a declarative approach! It is an indictment of YAML.
> Configuration is code not data.
Data > code. Data does not need to be debugged. The best code you can have is deleted code - deleted code does not need to be maintained, updated, or patched. Code is a necessary evil we write in order to build operable systems, not a virtue in and of itself.
local webdir = "/www/site/htdocs"
templates =
{
{
template = "html/regular",
output = webdir .. "/index.html",
items = "7d",
reverse = true
},
{
template = "rss",
output = webdir .. "/index.rss",
items = 15,
reverse = true
},
{
template = "atom",
output = webdir .. "/index.atom",
items = 15,
reverse = true
},
}
When I reference the configuration state, templates[1].output will be "/www/site/htdocs/index.html". And if the base directory changes, I only have to change it in one location, and not three.Allowing computation can alleviate some of those problems, whether it's done "declaratively" (e.g. prolog-like, as in cue) or not (e.g. like idealised algol with memory cells).
The main reason to avoid jumping to something like Python isn't that it's "not declarative"; it's that Python is impure, and hence may give different results on each run (depending on external state, random number generators, etc.). Python can also perform arbitrary external effects, like deleting files, which is another manifestation of impurity that we'd generally like to avoid in config.
tl;dr The problem isn't the style of computation, it's the available primitives. Don't add non-deterministic or externally-visible effects to the language, and it wouldn't really matter to me whether it's "declarative" or not.
That data you are talking about does need to be debugged, like Helm charts and pipeline definitions. Sure data is better, but config is code, not data.
Whether the generated declarative output produces the expected behavior on the part of the tool interpreting the declarative output is part of the tool's contract, not the generator or the declarative output. If you need to check the tool's behavior then either a) you wrote the tool or b) you're writing an acceptance test for the tool, which is an entirely different endeavor.
There's nothing that prevents you from writing a unit test that runs `helm template` directly to check whether a given chart with given values will produce a given set of YAML files.
Config is both. Config variables are data. The code that accesses and uses those variables is...well...code. they should be kept separate. Like any other code and data. Config isn't a separate special entity, it's just another part of the program. The data part should be represented as such and the code part should be code. Trying to combine them and create a special 'config' language is mistake.
These are people that by and large don't want to code, not saying that they can't or won't.
To be fair this has been in Windows shops, where scripting has only recently (last 5-10 years) taken off, so you've got a lot of windows admins that the closest they've been to code is Batch scripting with a bit of Powershell. This is a big change for them
As it happens i read about pulumi recently and I've put it on mt list of todo things, but I can't see that I'll be able to sell it to our team and our team is blessed (cursed?) with three former developers
I disagree. That's the stereotype that tool builders have of such people. Good ops people have always loved coding, or we wouldn't be living on the mountains of Bash scripts also known as "Linux distributions".
(Besides, people who don't like to code won't like writing tons of declarative markup either. So there is little point in the current approach either way.)
As to the declarative markup, for instance Azure Devops still doesn't have feature parity between YAML pipelines and classic pipelines, so while you could well be right about the same resistance to yaml, it's not, necessarily an issue yet.
I'm on a relatively new team kind of like that (it's not a former Ops team but most of the team was pulled from Ops/DBA/analysis teams, and fits the description, and it's not a really a DevOps but more of a Dev+Ops team) and the general reaction of the team to being introduceed to the AWS CDK has been “we need to move to that as soon as we can”, even from people who very vocally never wanted to be programmers. And that's with, in many cases, a couple months experience with both programming and YAML IAC in the form of CloudFormation.
The more things change, the more they stay the same.
The pendulum is swinging back towards scripting languages. But give it a few years, and we'll be railing against scripting languages for not being idempotent enough, and moving back to the latest version of YAML and XML (perhaps we'll go with TOML this time) with custom interpreters.
We have seen this half a dozen times already, between turing complete DSLs in ruby, to plain bash, perl, and python scripts. The other side are the DSLs written using YAML, JSON, and XML (and every other config language written in history).
I've only really skimmed their documentation, but the idea that calling 'new' against a VM class instantiates a VM in production seems to be to magical for my taste, and might be an indication that it's yet another product that targets the happy path more than what we tend to experience in production: failures.
As any Lisper knows, code is data is code.
Could you share some details on that? A quick Google search didn't reveal anything about major changes to the SDK.
Language doesn't matter. What matters is the attitude of tool builders that everything should "simply" be described as data. When that turns out to be insufficient, as it inevitably does, hacks are introduced and sooner or later you end up with crippled and idiosyncratic pseudo-languages.
The main culprits are Ansible/Salt and all the string-templating based tools for Kubernetes (Helm/Kustomize/...).
Especially with tools like Helm I believe we reached peak insanity levels. Instead of using a general purpose or configuration-specific (like Jsonnet/Cue/Dhall) programming language to build and then emit in-memory objects to YAML manifests, the industry is using YAML to define templated YAML and then parametrize this templating with even more YAML.
Now we just write complex declarative configuration with no debugging facilities, a non-existent type system, inversion of control that usually ends up working against us and no way to interact with external data apart from stringing everything together with fragile shell scripts. But hey, there's no need to write a single line of Go/Python/Java/Ruby!