It's 2022 and we can't even get a plaintext configuration format from 1980 right.

To me, it's more depressing that we've been at this for 50-60 years and still seemingly don't have an unambiguously good plaintext configuration format at all.

The difference between a data format and a configuration file is the use case. JSON and YAML were invented to serialize data. They only make sense if they're only ever written programmatically and expressing very specific data, as they're full of features specific to loading and transforming data types, and aren't designed to make it easy for humans to express application-specific logic. Editing them by hand is like walking a gauntlet blindfolded, and then there's the implementation differences due to all the subtle complexity.

Apache, Nginx, X11, RPM, SSHD, Terraform, and other programs have configuration files designed by humans for humans. They make it easy to accomplish tasks specific to those programs. You wouldn't use an INI file to configure Apache, and you wouldn't use an Apache config to build an RPM package. Terraform may need a ton of custom logic and functions, but X11 doesn't (Terraform actually has 2 configuration formats and a data serialization format, and Packer HCL is different than Terraform HCL). Config formats minimize footguns by being intuitive, matching application use case, and avoiding problematic syntax (if designed well). And you'd never use any of them to serialize data. Their design makes the programs more or less complex; they can avoid complexity by supporting totally random syntax for one weird edge case. Design decisions are just as important in keeping complexity down as in keeping good UX.

Somebody could take an inventory of every configuration format in existence, matrix their properties, come up with a couple categories of config files, and then plop down 3 or 4 standards. My guess is there's multiple levels of configuration complexity (INI -> "Unixy" (sudoers, logrotate) -> Apache -> HCL) depending on the app's uses. But that's a lot of work, and I'm not volunteering...

It has a good balance between expressivity and readability, it got enough logic to be useful, but not so much it begs for abuses, it can import/export to yaml and json and features an elegant type system which lets you define both the schema and the data itself.

I hope it gains traction.

Although I do feel like there is a case to be made that if you need a Turing complete configuration language then in most cases you failed your users by pushing too many decisions on to them instead of deciding on sensible defaults.

And if you are dealing with one of the rare cases where Turing complete configuration is desirable then maybe use Lua or something like that instead.

[1] https://hitchdev.com/strictyaml/

[2] https://github.com/crdoconnor/strictyaml

People create config languages that work for their use case and then it is just a happy accident if it works for other things.

I don't think anyone has put serous effort into designing a configuration language. And by that I mean collect use cases, study how other config languages does things, make drafts, and test them. etc...

I know a lot of people hate it but I find it to be the only configuration language that makes any sense for moderately large configs.

It’s short, readable, unambiguous, great IDE support. Got built in logic, variables, templates, functions and references to other resources - without being Turing complete imperative language, and without becoming a xml monstrosity.

Seriously there is nothing even close to it. Tell me one reasonable alternative in wide use that’s not just some preprocessor bolted onto yaml, like Helm charts or Ansible jinja templates.

I will take a kubernetes deployment manifest as an example that you would want to express in a hypothetically perfect configuration language. Now, eventueally, you end up in the "containers" bit of the pod template inside the deployment spec.

And in that, you can (and arguably should) set resources. But, in an ideal world, when you set a CPU request (or, possibly, limit, but I will go with request for now) for an image that has a Go binary in it, you probably also want to have a "GOMAXPROCS" environment variable added that is the ceiling of your CPU allocation. And if you add a memory limit, and the image has a Java binary in it, you probably want to add a few of the Java memory-tuning flags.

And it is actually REALLY important that you don't repeat yourself here. In the small, it's fine, but if you end up in a position where you need to provide more, or less, RAM or CPU, on short notice (because after all, configuration files drive what you have in production, and mutating configuration is how you solve problems at speed, when you have an outage), any "you have to carefully put the identical thing in multiple places" is exactly how you end up with shit not fixing themselves.

So, yeah, as much hate as it gets, BCL may genuinely be better than every other configuration language I have had the misfortune to work with. And one of the things I looked forward to, when I left the G, was to never ever in my life have to see or think about BCL ever again. And then I saw what the world at large are content with. It is bloody depressing is what it is.

https://github.com/dhall-lang/dhall-lang

https://cuelang.org/docs/usecases/configuration/

Me, the programmer finds those kinda cool.

And the sysadmin in me developed a dislike of both within 1 minute of looking at them.

Honestly, I think a good configuration library should be more than a spec, it should come with a library that handles parsing/validation. See, there are two sides to configuration, the user and the program. Knowledge about the values, defaults and types should live on the program side and should be documented. Then the user side of configuration can be clean and easy to read/write and most important of all, allow the user to accomplish the most common configuration without having to learn a new config language on top of learning the application.

You just described CUELang.

The type system allows to define a schema as well as the data, in the same file, or in 2 separate ones. Then you can call either a cli tool (that works on linux, windows or mac) or use the Go lib (or bind to it).

For compat, cue can import and export to yaml, json and protobuf, as well as validate them.

I think it's the opposite. There isn't a single config file that suits all needs.

Especially when you realize config isn't a single thing.

http://mikehadlow.blogspot.com/2012/05/configuration-complex...

I haven’t studied it, I am just generally feeling unhappy about most software configuration.

Ansible is worse than Puppet and CFEngine in many ways, but it is superior in the user interface.

It managed to not only be a config management solution, but provide a universal config language that most apps could be configured with. So for a lot of use cases, if you know Anisible/YAML then you don't have to learn a new configuration language on top of learning a new application.

In any case, I found Ansible scripts to have like a 3 month half life. If we were lucky. I'm not bitter.

My dream configuration system should revert to default when the config is removed (keeping data). Have a simple/easy user interface. Have maintained modules with sane defaults for the 500 most common server software. I would rather there be no module than an abandoned one with unsafe defaults, that way it is clear that I would have to maintain my own if I want to use that particular piece of software. Performant, it really shouldn't take more than a few minutes to apply a config change. No more than 30 min for initial run.

GitHub.com/vitiral/zoa

It has both binary and textual representation (with the first byte being able to distinguish them), and the syntax is clean enough I'm planning on extending it into a markup language as well.

- it has a spec

- it has other types than strings

- you can always decide you actually need nested data, and add them later

Seriously: for application-specific config files, the lack of a formal spec can be kind of a nice thing. You can design your parser to the exact needs of your program, with data types that makes sense for your use case. Throw together a formal grammar for use in regression testing, and you're all set.

Obviously a formal spec is essential for data interchange, but that's why JSON exists. To me, YAML is in a gray area that doesn't need to exist. The same thing goes for TOML, but to a far lesser extent.

But isn't the config file just a string?

https://github.com/toml-lang/toml/issues/516

The official way to do list of tables is (look at how much duplication there is)

  [[main_app.general_settings.logging.handlers]]
    name = "default"
    output = "stdout"
    level = "info"

  [[main_app.general_settings.logging.handlers]]
    name = "stderr"
    output = "stderr"
    level = "error"

  [[main_app.general_settings.logging.handlers]]
    name = "access"
    output = "/var/log/access.log"
    level = "info"

  handlers = [
    {
      name = "default",
      output =  "stdout",
      level = "info",
    }, {
      name = "stderr",
      output =  "stderr",
      level = "error",
    }, {
      name = "access",
      output =  "/var/log/access.log",
      level = "info",
    },
  ]

When configuration gets so complicated that the configuration starts to resemble structured data I tend to prefer to switch to a real scripting language and generate JSON instead.

(someone may point out to me a CI platform that relies on TOML—which I welcome)

If you need to program complex logic to build a crate, you don’t write TOML. You write a build.rs file in actual Rust.

JSON5?

https://github.com/python/cpython/blame/d75a51bea3c2442f81d3...

Oh, maybe it’s this issue:

https://bugs.python.org/issue34132

If I’ve read it correctly, there was a regression from Python 2.x to 3.x such that you now need to format comments:

    #like this 

    # like this

    try:
        try:
            import orjson as json
        except:
            try:
                import rapidjson as json
            except:
                try:
                    import fast_json as json
                except:
                    import json
        foo = json.loads(string)
    except:
        try:
            import yaml
        except:
            # try harder
            import os
            try:
                assert(os.system("pip3 install yaml") == 0)
            except:
                # try even harder
                try:
                    assert(os.system("sudo apt install python3-pip && pip3 install yaml") == 0)
                except:
                    assert(os.system("sudo yum install python3-pip && pip3 install yaml") == 0)
            import yaml
        try:
            foo = yaml.loads(string)
        except:
            try:
                ....

  pip install --user yaml

I've seen that kind of approach cause a ton of issues the moment that the software was used in a different environment than the author expected.

It's much better IMO to fail with a message about how to install the missing dependency.

Considering we are having software drive cars today it should be trivial and I would say even arguably expected that software should be able to autonomously "figure out" how to run itself and avoid conflicts with other software since that's a trivial task in comparison to navigating city streets.

    PATH=$HOMEBREW_PREFIX/opt/ansible/libexec/bin:$PATH
    pip list | grep -i yaml
    python -V
    python <<'DOIT'
    from io import StringIO
    import yaml
    print(yaml.safe_load(StringIO(
    '''
        {
            "list": [
                {},
                    {}
            ]
        }
    ''')))
    DOIT

    PyYAML                6.0
    Python 3.10.1
    {'list': [{}, {}]}

  $ sed 's/\t/--->/g' break-yaml.json
  --->{
  --->--->"list": [
  --->--->--->{},
  --->--->--->{}
  --->--->]
  --->}
  $ jq -c . break-yaml.json
  {"list":[{},{}]}
  $ yaml-to-json.py break-yaml.json
  ERROR: break-yaml.json could not be parsed
  while scanning for the next token
  found character '\t' that cannot start any token
    in "break-yaml.json", line 1, column 1
  $ sed 's/\t/    /g' break-yaml.json | yaml-to-json.py
  {"list": [{}, {}]}

YAML does not allow tabs in indentation, but the tabs in your example are not indentation according to the YAML spec productions.

You can see it clearly here against many YAML parsers: https://play.yaml.io/main/parser?input=CXsKCQkibGlzdCI6IFsKC...

As tinita points out, sadly PyYAML and libyaml implement this wrong.

See https://matrix.yaml.info/

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... says:

> Insignificant whitespace may be present anywhere except within a JSONNumber [forbidden] or JSONString [interpreted as part of the string]

And specifically lists tab as whitespace:

> The tab character (U+0009), carriage return (U+000D), line feed (U+000A), and space (U+0020) characters are the only valid whitespace characters.

More specifically, expanding https://datatracker.ietf.org/doc/html/rfc8259#section-2 gives an array as (roughly)

> ws %x5B ws value (ws %x2C ws value)* ws %x5D ws

Where `ws` explicitly includes `%x09`. Which seems to cover this case?

      ws = *(
              %x20 /              ; Space
              %x09 /              ; Horizontal tab
              %x0A /              ; Line feed or New line
              %x0D )              ; Carriage return

  json
    element

  element
    ws value ws

  ws
    ‘0009’ ws

The error from PyYaml 5.3.1:

    yaml.scanner.ScannerError: while scanning for the next token
    found character '\t' that cannot start any token
      in "<unicode string>", line 4, column 1

  $ printf '{\n\t"list": [\n\t\t{},\n\t\t{}\n\t]\n}\n' > test.json

  $ jq < test.json 
  {
    "list": [
      {},
      {}
    ]
  }

  $ yamllint test.json 
  test.json
    2:1       error    syntax error: found character '\t' that cannot start any token (syntax)

It would be great if instead of the histrionic message on CPAN (which amusingly accuses others of "mass hysteria"), the author would just say "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML".

The YAML spec should be updated to reflect this, but I wonder if a simple practical workaround in YAML parsers (like replacing each tab at the beginning of the document with two spaces before feeding it to the tokenizer) would be sufficient in the short term.

But YAML can start with tabs. Tabs are allowed as separating whitespace in most of the spec productions but are not allowed as indentation. Even though those tabs look like indentation, the spec productions don't interpret them as such.

See my comment above and esp see https://play.yaml.io/main/parser?input=CXsKCQkibGlzdCI6IFsKC...

Note: the YAML spec maintainers (I am one) have identified many issues with YAML which we are actively working on, but (somewhat surprisingly) we have yet to find a case where valid JSON is invalid YAML 1.2.

Speaking of PyYAML, I recently ran into an issue where I had to heavily patch PyYAML to prevent its parse result from being susceptible to entity expansion attacks. It would be nice to at least have a PyYAML mode to completely ignore anchors and aliases (as well as tags) using simple keyword arguments. Protection against entity expansion abuse would be nice too.