News for hackers. Stuff that matters.
Re:YAML: It's Time to Move On by thefifthsetpin on Sunday November 14, 2021 @05:26PM (#29221323)
I don't like YAML and would like to move on, but I hope we don't move onto this.

I think it's crazy that when I add a string to an inline list, I may need to convert that inline list to a list because this string needs different handling. I think it's crazy that "convert an inline list to a list" is a coherent statement, but that is the nomenclature that they chose.

I don't like that a truncated document is a complete and valid document.

But what is most unappealing is their whitespace handling. I couldn't even figure out how to encode a string with CR line endings. So, I downloaded their python client to see how it did it. Turns out, they couldn't figure it out either:

>>> nt.loads(nt.dumps("\r"),top="str") '\n'

Re:YAML: It's Time to Move On by nanis on Sunday November 14, 2021 @08:11PM (#29222425)
I wish people would stop trying to write programs for which there are no interpreters, compilers, or linters:

    name: Install dependencies
    run:
        > python -m pip install --upgrade pip
        > pip install pytest
        > if [ -f 'requirements.txt' ]; then pip install -r requirements.txt; fi
That is a program that is hiding in the bowels of a "nestedtext" document ... It is no better than a program that is hiding in the bowels a JSON or YAML document.

We all have to deal with this, but it is beyond stupid.

    [Install Dependencies]
    run=/path/to/install-script
Then, write `install-script` in whatever language you want ... verify it works. It should have tests. etc etc etc.
Re:YAML: It's Time to Move On by theamk on Sunday November 14, 2021 @09:37PM (#29223007)
I don't think it matters much if this is inline or in separate file. If you want to test your tests, "yq -r .run input.yaml | sh -e" works as well.

In fact, if I really wanted to test my tests, I'd say that directly testing the corresponding clause is the more comprehensive approach. For example, what if someone accidentally changes the line to read:

    run=/path/to/install-scriptq
? then your test of "install-script" will not catch anything. But if your test runs "yq -r .run | sh -e", then it will catch that error. And you can still forward to a script if you wanted to.

So let's keep inline scripts, they are very reasonable methods for just a few commands.

Re:YAML: It's Time to Move On by galkk on Monday November 15, 2021 @06:19AM (#29226127)
Depending on the source control tool you may lose syntax highlightning, you, most likely, lose linters and even copying those multi-line commands to shell becomes cumbersome. I consider inlining example from GP's comment awful
Re:YAML: It's Time to Move On by kitd on Monday November 15, 2021 @01:31AM (#29224422)
V minor point: new yq versions use 'e' instead of '-r'.
Re:YAML: It's Time to Move On by ljm on Sunday November 14, 2021 @06:29PM (#29221781)
It would be nice if YAML wasn't horrendously abused the way it is. You have CI pipelines that let you construct DAGs to represent your builds, but you need several thousand lines of YAML and a load of custom parsing to get programming constructs in the string types, for example. And then each provider has its own way of providing those.

I don't have to re-read manuals describing how to do if/else in Ruby or Java or Lisp, but as soon as yaml and some 'devops' tooling is involved, I have to constantly jump back and forth between the reference and my config.

The main point being that the problem isn't the file format but the products that continue to push it, presumably because hacking stuff on top of `YAML.parse` is less effort than designing something that fits the purpose.

Re:YAML: It's Time to Move On by donmcronald on Sunday November 14, 2021 @07:35PM (#29222181)
Yeah. A lot of times I find myself thinking YAML is like a really awful programming language. You can sort of do conditional logic and loops, but usually I find it hard to follow what's going on.

For build systems, I always liked the idea of Gradle where the core functionality was simple and declarative, but with the option to use a real programming language for things that weren't simple. For example, integrating installers or form builders (pre-processing) into a build are things I would consider non-trivial if there aren't official plugins, but it was still relatively easy to do with Gradle.

The biggest problem I always had with Gradle was that I didn't like Groovy and I always though there was a missed opportunity to have a statically typed build system with a solid API/contract and all the fancy tooling like auto-complete that you get with statically typed languages.

I see JSON5 mentioned a lot in the comments. In terms of CI / build systems, I feel like something built with JSON5/TypeScript could be really good. I'd be really happy using TypeScript for configuring things like build systems where there shouldn't really be an argument for needing it to be usable by non-programmers.

Personally I feel like I've spent way to much of my life debugging YAML syntax issues.

Re:YAML: It's Time to Move On by kitd on Monday November 15, 2021 @01:36AM (#29224452)
If you're happy to go lispy, there's Babashka [1], a Clojure without the JVM. It has built-in support for 'tasks' designed to make writing build scripts easy.

[1] https://babashka.org/

Re:YAML: It's Time to Move On by TheCycoONE on Sunday November 14, 2021 @10:54PM (#29223492)
Does the Kotlin support in Gradle solve your problem? https://docs.gradle.org/current/userguide/kotlin_dsl.html
Re:YAML: It's Time to Move On by spuz on Monday November 15, 2021 @05:45AM (#29225893)
My experience with Kotlin gradle scripts is worse than Groovy. For example, given the following valid groovy/kotlin gradle program:

    dependencies {
    }
What would you expect to see between the curly braces? IntelliJ IDEA which supposedly has full support for the gradle DSL both for Groovy and Kotlin offers only generic suggestions. Common function calls such "implementation()" or "testImplementation()" are not suggested. If you do use those functions, no suggestion is made for their parameters. Because Gradle's DSL is built on top of a general purpose language, it loses the benefits of a DSL (constraining the set of possible configurations and guiding the user towards valid configurations).
Re:YAML: It's Time to Move On by netvl on Tuesday November 16, 2021 @01:14PM (#29244233)
The key benefit of the Kotlin DSL is that in this precise example, IDEA does suggest valid stuff: https://imgur.com/a/vFYNIU1

Kotlin DSL is miles ahead of Groovy in terms of discoverability and IDEA integration. With Groovy DSL, most of the build script is highlighted with various degrees of errors and warnings; with Kotlin DSL, if something is highlighted, it is a legitimate error, and vice versa - if no errors are detected by IDEA, then it is almost certain to work.

There were rough spots of IDEA integration a couple years ago, but now it is close to perfect, within Gradle's limits of course (due to sheer dynamic nature of it, some things are just not possible to express in a static fashion, unfortunately). The biggest obstacle to Kotlin DSL use might be that some of the plugins use various Groovy-specifc features which are hard to use from Kotlin, but thankfully most of the plugins either fix those, or are rewritten in Java or Kotlin instead.

Re:YAML: It's Time to Move On by heurisko on Monday November 15, 2021 @09:09AM (#29227537)
This was one thing I found difficult learning Gradle: its seeming complete lack of autodiscoverability.

I expected it to catch on, but I think a lot of people are sticking with Maven.

Re:YAML: It's Time to Move On by spuz on Monday November 15, 2021 @09:45AM (#29227972)
There's a huge gap in Java build tool space for a tool that is simple and easy to learn and can cover 90% of projects' requirements. I have this feeling that we're in the "subversion" days of java build tools and the day someone introduces "git" people will wonder why we suffered with Gradle and Maven for so long. If I had time I would be looking into building this.
Re:YAML: It's Time to Move On by zmmmmm on Wednesday November 17, 2021 @05:30AM (#29251204)
predating Gradle was a tool called gant. It was simple, intuitive and did 90% of what every project could want. Ironically it was Groovy based as well. But instead of the Gradle arcane magic based configuration it was literal, direct, a simple extension of Ant that came before it. I liked it much better, but someone decided they could make a business out of Gradle and gant got deprecated and here we are.
Re:YAML: It's Time to Move On by ljm on Monday November 15, 2021 @05:39AM (#29225863)
I found it fairly simple to build Gradle plugins with Kotlin. If anything, the problem was just having the patience to actually find the right documentation in the first place, and understand what was being described. The main problem I faced there was that I wanted a plugin to configure dependencies for the project it would run against and the docs around dealing with dependencies and detached configurations were a bit confusing.

I do find it curious that a lot of these tools get seen as basic task runners despite offering much more potential.

Re:YAML: It's Time to Move On by ilyash on Monday November 15, 2021 @02:11AM (#29224651)
It's always the same trajectory with declarative programming. It starts with "it's just configuration, we need something simple". Then users come with use cases which are more complex. Then you have programming language on top of configuration language syntax.

* https://ilya-sher.org/2018/06/30/terraform-becomes-a-program...

* https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a...

Re:YAML: It's Time to Move On by stevekemp on Monday November 15, 2021 @08:12AM (#29226922)
Very much so. A good few years ago I got annoyed that I couldn't have change mutt configuration the way that I wanted, because it has a built in configuration language which doesn't allow complicated conditionals etc.

(There are workarounds, and off-hand I can't think of a great example, but bear with me.)

In the end I wrote a simple console-based mail-client, which used a Lua configuration file. That would generally default to using hashes, and key=value settings, but over the time I used it things got really quite configurable via user-defined callbacks, and functions to return various settings.

For example I wrote a hook called `on_reply_to`, and if you defined that function in your configuration file it would be invoked when you triggered the Reply function. This kind of flexibility was very self-consistent, and easy to add using an embedded real language.

Later I added some hacks to a local fork of GNU Screen, there I just said:

* If the ~/.screenrc file is executable, then execute it, and parse the output.

That let me say "If hostname == foo; do this ; otherwise do this .." and get conditionals and some other things easily. Another example was unbinding all keys, and then only allowing some actions to be bound. (I later submitted "unbindall" upstream, to remove the need for that.)

Re:YAML: It's Time to Move On by silon42 on Monday November 15, 2021 @03:23AM (#29225039)
Some even start with a programming language and pretend it's declarative...

IMO, it's only declarative when it's a data model which is easily parsed by multiple languages/systems where it's needed.

Re:YAML: It's Time to Move On by int_19h on Sunday November 14, 2021 @10:25PM (#29223305)
What's really sad is that XML had a much better ecosystem around this for ages. I'd very much rather deal with XQuery or even XSLT to construct build trees, than the current crop of ad-hoc YAML preprocessors. At least the XML stuff had a consistent type system underneath!
Re:YAML: It's Time to Move On by tdeck on Sunday November 14, 2021 @10:33PM (#29223362)
XSLT is an absolute horror and not something I would want to deal with again. It feels like some weird academic experiment in an XML declarative programming language that should never have made it to print.

If something needs the flexibility of a programming language, why not use a real one that's been well tested for writing other programs? These various config file programming systems always end up creating something notorious that everyone tries to avoid having to work on.

Re:YAML: It's Time to Move On by int_19h on Monday November 15, 2021 @02:54PM (#29231860)
XQuery is, in many ways, XSLT with better syntax. It doesn't have the pattern-matching transforms that are the T in XSLT - but for configs, I don't think it makes a big difference.

Also, I don't think many realize that the stack has evolved since early 00s. XSLT 1.0 was a very limiting language, requiring extensions for many advanced scenarios. But there's XSLT v3.0 these days, and XPath & XQuery v3.1, with some major new features - e.g. maps and lambdas. Granted, this doesn't fix the most basic complaint about XSLT - its insanely verbose syntax - but even then, I'd still take XSLT over ad-hoc YAML-based loops and conditionals.

Re:YAML: It's Time to Move On by stjohnswarts on Monday November 15, 2021 @06:24PM (#29234335)
I will take the verbosity of XML any day over YAML wrestling (complex YAML configs of course). There is simply too many "implicit rules" for YAML. it's why I prefer python over ruby and perl. Generally though TOML has been good enough for me to do lots of fairly large config files that are easy for humans and machines to parse.
Re:YAML: It's Time to Move On by dnautics on Sunday November 14, 2021 @10:36PM (#29223382)
XML died because too many configurations turned what should be a 'prop' into an inner tag -- and it doesn't help that XML doesn't really give guidance as to when to use which. And, of course, when you deserialize XML, the innerText is always in a very strange place as to not really being clear "what the right way to handle it" is.
Re:YAML: It's Time to Move On by thayne on Monday November 15, 2021 @12:28AM (#29224057)
Honestly, I think using an embedded scripting language, like lua or even javascript, would be a much better fit for these use cases than trying to make yaml do something it wasn't designed for.
Re:YAML: It's Time to Move On by ljm on Monday November 15, 2021 @05:43AM (#29225881)
Ironically, having used cdk8s[1] for dealing with kubernetes infrastructure, that's the one thing where I've actually preferred yaml. That said, k8s resource definitions are pure config so there's no need to try and hack extra bits on top of a serialized data structure.

[1]https://cdk8s.io/

Re:YAML: It's Time to Move On by theamk on Sunday November 14, 2021 @09:40PM (#29223018)
I really like approach of buildkite CI -- they use yaml, but this yaml can be produced by an executable script.

So you write yaml by hand for trivial cases, but once it get complex, you can just drop back to shell/python/ruby/node/whatever, implement any complex logic, and serialize results to plain yaml.

Re:YAML: It's Time to Move On by shusaku on Monday November 15, 2021 @02:19AM (#29224706)
This is the crux of the issue right? YAML is fine for most projects, but THE project for using YAML is CI configuration, for which it doesn’t match.
Re:YAML: It's Time to Move On by Lhiw on Sunday November 14, 2021 @06:40PM (#29221852)
I'm pretty sure the format is the issue.

I still don't know how to do arrays in yaml.

Is it new line, tab to same indent or same indent + one space, do I need a dash? Does a dash make it an array or an object?

It's just simply not that big of a deal to add a few quotes and braces to make everything make sense.

The only real issue with Json is the lack of comments and strictness about extra commas.

Re:YAML: It's Time to Move On by tialaramex on Sunday November 14, 2021 @07:27PM (#29222130)
> I don't like that a truncated document is a complete and valid document.

Me either. If your documents have this property you're likely to tempt people to start trying to process partial documents.

When they do that, they violate Full Recognition Before Processing and likely there's a latent security bug as a result.

Re:YAML: It's Time to Move On by jrochkind1 on Sunday November 14, 2021 @07:00PM (#29221974)
> So, I downloaded their python client to see how it did it.

Who are you suggesting the python client belongs to, who is 'they' in 'their'?

Re:YAML: It's Time to Move On by detaro on Sunday November 14, 2021 @07:16PM (#29222068)
... the people that made it and wrote the submission we are discussing? how is this even a question?