That said, I hope waf and scons die off soon (only a handful of packages even use them). CMake and Meson are much better choices.
Why not? In the old days, it was "./configure; make". Today, it's much less certain what build system(s) you'll have to deal with when building a FOSS tool. That's not a problem when everything works, but as experience shows, more often than not you'll run into issues and then I would say it's nice if you are familiar with the system.
And just like with other tools it's nice if everybody uses the same thing. Github wouldn't exist if everybody rolled their own version control system, for example.
It also wouldn't exist if no one did. It's all about compromises. Having too many tools for the same job isn't good, but monoculture isn't either.
Sadly when it comes to C++ I don't think we'll get a do-over. Too many build systems exist already, and important libraries from a decade or two ago will use those systems until the end of time, and so will the projects that depend on them indirectly.
I haven’t tried Meson, but CMake cant really do much besides compile the programming languages it explicitly supports. Automation tasks are pretty much impossible.
I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. Only having to know a half dozen build systems is way easier than having to know waf, Scons, meson, autotools, make, setuptools, cargo, qmake, cmake is a pain.
Only 25 packages in the distro I contribute to use waf. Most are audio recording tools that do not get much maintenance. 20 packages use scons. Meanwhile almost 400 use meson, ~500 use qmake, ~1000 use cmake, and over 2000 use autotools.
Not currently, but I do have a half-written article I plan to publish eventually once I complete this project I'm working on, to showcase my use of waf and how it's helped me. That's how much I like it!
But to give you a quick example: my project (a cross-platform game) has a bunch of assets that I am embedding in the executable. To do this, I wrote a custom waf task that automatically generates a char array in a .cpp/.h bundle, and adds it to my build and include paths.
Using that same tool, I am able to embed other assets that need to be transformed in various ways before being embedded. One of those is a bundle of binary files and meta-data combined together inside a single flatbuffer data blob. This involves compiling the schema, generating a small command-line utility to write the blob using the sources generated from the schema compiler, using that utility to generate the blob, and finally embedding the blob using the asset embedding tool. (Plus some of the binary files themselves need to be processed before all of this happens)
I could do that with shell scripts, but waf's design makes it easy to do, in a small amount of reusable code (everything I just described I implemented in <400 lines of python), and in a general-purpose and easy to read programming language. Plus the tasks automatically execute in parallel wherever possible.
> I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects.
I've heard that complaint before, and I can see why it would be annoying for outside maintainers. Waf's customizability means two Waf projects might look nothing alike. So even if one becomes an expert at Waf (which is not very easy considering how bad the docs are), they might be completely lost in someone else's project. I failed a game jam once because I was using Amazon Lumberyard (which uses waf), and even though I completed the game, I couldn't figure out how to compile the damn thing before the deadline.
But if it's any consolation, I don't plan on distributing my project through any linux distros :P
Core primitives, like 'a file is out of date iff its modification time or hash has changed', should be expressible 'in userspace', as it were—the core of the build system shouldn't know about them. (Though it should have some concept of a target being up-to-date and not being rebuilt, it shouldn't need to know about files.) This, combined with the ability to create proper abstractions, means you can emulate the primitives of all existing build systems and give them a common substrate.
(Bonus: assuming the core primitives are done right, you can still get performance as good as ninja. Which is good, because you won't be able to compile to a build system like ninja.)
Shake is written in Haskell as a library (so your build rules have to be written in Haskell too, as programs) but you could either wire up a language via the FFI to make some classes of build rules easily expressible in JavaScript/Lua, or in theory rewrite the core of Shake in another language too. The theory is all there, though.
My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity (multi language builds, requires parsing or other "advanced" features, etc). As a bonus it's also very fast, within a few % of Ninja when parsing Ninja files, and sometimes faster than that. I was surprised when it came in as 2x as fast as Ninja on a large-ish C codebase at work, due to having picked a different "build schedule" that distributed compilations across cores better.
> A target is a file we want the build system to produce
This is somewhat arbitrary, and shouldn't be mandated by the build system. You say 'the secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files', but that doesn't seem to be corroborated?
They do show examples of 'phony' targets, but why not simply eliminate the conflation of build targets and files entirely? Phony targets are just a holdover from make.
______________________________________________________
Systems evolve. You don't want to rewrite your build scripts the moment you cross the complexity line.
Your mention of Shake is nice though. However, I'm not sure if it could gain widespread adoption, since it's based on Haskell which is not very popular unfortunately (and has a bad name when it comes to learning curve).
There's very few build systems that actually support this in a reproducible and hermetic way.