News for hackers. Stuff that matters.
Re:Meson 1.0 Build System by tristan957 on Friday December 23, 2022 @05:40PM (#34111411)
If the only reason to use CMake is user-defined functions, I see no real reason to use it. The gains in readability of Meson are far and beyond the gains of user-defined functions in my opinion. I maintain a large open source library and haven't once felt the need for user-defined functions.
Re:Meson 1.0 Build System by oldgradstudent on Friday December 23, 2022 @05:50PM (#34111497)
The excuse is still wrong, and indicates complete misunderstanding of what Turing-complete means.

I write plugins for a large, well known application.

Building the plugin is a complicated multi stage process with many dependencies.

I've spent a lot of time understanding the process (undocumented, only VS and Xcode project files are provided in the SDK).

I now have a single function call to create each plugin.

Re:Meson 1.0 Build System by tristan957 on Friday December 23, 2022 @05:55PM (#34111554)
What is hard about the following?

    # project/meson.build
    project_plugin_dep = declare_dependency(
        include_directories: plugin_includes,
        compile_args: plugin_c_args
    )

    # plugin/meson.build, repeated for each plugin
    shared_module(
        'plugin',
        'plugin.c',
        dependencies: project_plugin_dep
    )
Without having seen what your user-defined function does, this is the best equivalent that I can come up with. If your plugins exist out of tree, then Meson can consume a pkg-config file. Postgres is slowly moving to Meson and uses out of tree plugins.
Re:Meson 1.0 Build System by oldgradstudent on Friday December 23, 2022 @06:11PM (#34111709)
I also need to generate a file, pass it through a special compiler, pass the result through the system resource compiler, link it with the shared library.

There are a bunch of other steps. I have a YAML description of the plugin which I use to generate plugin skeleton, and so on.

Do you really think it makes sense to repeat that again and again?

And then, what if I need to fix a bug in the process?

Re:Meson 1.0 Build System by bonzini on Friday December 23, 2022 @06:26PM (#34111865)
Can you use a foreach loop?
Re:Meson 1.0 Build System by oldgradstudent on Friday December 23, 2022 @07:00PM (#34112178)
I'm sure I can use any number of ugly workarounds.

In many large projects, the build system is necessarily large and complicated. It should be treated like any other code, and would massively benefit from the most basic tool for writing clean, readable code - user-defined functions.

Sadly, as much as Meson is tempting in other respects, without user-defined functions, it is unfit for human consumption.

The excuse Meson uses, that user-defined functions would make it Turing-complete and therefore not statically analyzable is totally false.

Adding user-defined functions without recursion does not make the language Turing-complete. On the contrary, as long as there is no recursion, you can inline the functions and create an equivalent code without functions.

Re:Meson 1.0 Build System by tristan957 on Friday December 23, 2022 @07:28PM (#34112383)
> it is unfit for human consumption.

Surely this is hyperbole since some of the largest open source projects consider it a viable build system. Systemd, Mesa, GStreamer, GNOME, Postgres.

Would I be correct in assuming that you are coming at this conversation having never tried Meson?

Re:Meson 1.0 Build System by oldgradstudent on Friday December 23, 2022 @07:59PM (#34112602)
> Would I be correct in assuming that you are coming at this conversation having never tried Meson?

I tried to port the project I've previously mentioned here from CMake to Meson build, but could not find a way of doing it with Meson build in a satisfactory way.

The lack of user-defined functions is a show stopper because I have a complicated multis-stage build process. It is also very different between platform. Having to repeat parts of this process is disqualifying.

A programming language, and a build language is a programming language, without user-defined functions is unfit for human consumption.

Re:Meson 1.0 Build System by sfpotter on Friday December 23, 2022 @10:00PM (#34113444)
Can you use a script written in another programming language to generate your intermediate targets?
Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @05:28AM (#34115613)
Sure, I can write a script to do that.

The script can examine the dependencies, see if any of them changed and then rebuild only if needed.

It's like Greenspun's tenth law[1]:

> Any sufficiently complicated build script contains an ad hoc, informally-specified, bug-ridden, slow implementation of make.

But then, what do I need Meson for?

[1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

Re:Meson 1.0 Build System by sfpotter on Saturday December 24, 2022 @01:23PM (#34119542)
That's not what I meant. Of course, if you wanted to roll your own build system, you could do that. But hopefully you're aware that's not what I'm suggesting.

Meson provides custom targets, so Meson itself would examine the dependencies and rebuild if needed:

https://mesonbuild.com/Custom-build-targets.html

Obviously this might not do what you need specifically, but it does deal with some of the use cases for user-defined functions.

Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @04:24PM (#34121208)
Of course I have custom build scripts that build a target from its dependencies. The point is that there are several such targets as part of building a single plugin. It is the build system that decides whether to run such build script and in what order in order to build the final result, depending on which sources and intermediate targets were updated.

The entire point of having functions is to capture a "recipe" for building one type of target, and then be able to reuse it again and again.

Re:Meson 1.0 Build System by sfpotter on Saturday December 24, 2022 @06:24PM (#34122268)
Sounds like Meson isn't a very good match for your use case. That's too bad. Good thing there are other tools like CMake which are a better fit for you. It seems like you have also lost some time attempting to port your build over to Meson and failed, which was probably really frustrating. As a result, you really dislike Meson, going so far as to call it "unfit for human consumption".

Many other people like Meson, and many large projects have gone so far as to switch to using Meson. It's great that it worked for them, and it's OK that it didn't work for you.

Re:Meson 1.0 Build System by oldgradstudent on Sunday December 25, 2022 @10:06AM (#34127811)
> Sounds like Meson isn't a very good match for your use case.

That's exactly what I've been saying.

> As a result, you really dislike Meson, going so far as to call it "unfit for human consumption".

Meson got so many things right, but this one point makes it unusable for many projects, and prevents it from being industry standard.

Let me say it again, a programming language for writing large scale programs (11,000 lines is a large scale program) without functions/subroutines is unfit for human consumption.

It is made worse by giving a blatantly wrong rationale for not having user-defined function.

The only reason I even bothered spending so much time in responding is the hope that they will reverse this idiotic position.

Re:Meson 1.0 Build System by sfpotter on Sunday December 25, 2022 @01:53PM (#34129932)
I guess it wouldn’t help for me to recapitulate all of the arguments made by bonzini, who actually was involved in the 11,000 lines (down from 15,000 thanks to switching to Meson). He made his arguments very cogently and you apparently either don’t understand them or don’t care to listen.

Clearly there are people who think the position isn’t idiotic, and there are large projects which decided that Meson was a great choice and chose to migrate to it. This means that Meson is fit for human consumption.

Re:Meson 1.0 Build System by bonzini on Saturday December 24, 2022 @02:20AM (#34114773)
QEMU also has a complicated multi stage build process, and using foreach was not a workaround. Each directory sets up what it wants to build in a dictionary, and then the main meson.build does all the steps.
Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @12:55PM (#34119216)
So it's like having ad hoc, informally-specified, bug-ridden functions calls, just without argument checking, type checking, localized error reporting, etc?
Re:Meson 1.0 Build System by bonzini on Saturday December 24, 2022 @04:31PM (#34121261)
No, not really. It's just a different, data flow based approach.

The only thing I would use user-defined functions for is to structure the meson.build file in top-down manner, but each function would be called just once.

Re:Meson 1.0 Build System by camgunz on Saturday December 24, 2022 @05:16PM (#34121671)
Honestly, and I'm saying this with kindness, you are super hung up on Meson not having user-defined functions and their use of "Turing complete". Others have pointed out there are perfectly reasonable workarounds (external scripts and foreach loops aren't really that ugly), but you're just so miffed you refuse to consider them.

A lot of build systems won't support what you're asking here. Most don't do multi-stage builds, most don't have the concept of functions, many rely on external tools (autotools, pkg-config, etc). They're not "unfit for human consumption", which is pretty offensive to the people who chose Meson for their projects as well as Meson developers.

I get that you're offended by the situation. I might even agree with you in principle. But you're all over a Meson release thread, their 1.0 release thread even, shitting all over them, on Christmas Eve! Lighten up!

Re:Meson 1.0 Build System by palata on Friday December 23, 2022 @06:55PM (#34112139)
Does CMake have to generate the file and pass it through the special compiler? Can't it be done separately, before CMake is run?

One thing that I find make CMake configurations completely unreadable is to try to have it do everything, coffee included. Where often a script could be run separately first (e.g. to generate those files), and then CMake could run normally afterwards.

Re:Meson 1.0 Build System by oldgradstudent on Friday December 23, 2022 @07:05PM (#34112218)
> Does CMake have to generate the file and pass it through the special compiler? Can't it be done separately, before CMake is run?

No, it cannot be done before CMake runs. It is a target like any other build target. It depends on other files or targets, and should be rerun if any of them change, and if it is rebuilt, there are other targets that depend on it and should be rebuilt, and so on.

The entire point of the build system is to concisely and accurate specify the build graph, and only rebuild what is necessary.

Re:Meson 1.0 Build System by palata on Saturday December 24, 2022 @06:32AM (#34115932)
Ok let's take a concrete example. Say you have proto files (seems likely in a big system). Whenever you change a proto file, you know that you should run protoc, and then run CMake. If you add a proto file, not only you need to run protoc on this new file, but you need to tell CMake about the new generated file(s).

For the big majority of projects out there, it seems to me that it is perfectly fine to handle that manually, because projects are not that big. Do we agree on that?

Then of course, if you are a GAFAM with a huge monorepo, probably you need more advanced tools (they don't build their own tools for nothing). My feeling, though, is that most people complaining that CMake is crap actually work on small projects and can't be bothered to run protoc manually "because their philosophy is that it should all work with one button".

Re:Meson 1.0 Build System by plorkyeran on Saturday December 24, 2022 @12:16PM (#34118799)
No, I don't agree that having to manually run protoc after modifying a proto file is even slightly reasonable. That is exactly the sort of thing which the build system is there for, and the idea of not spending the few minutes it takes to automate that is baffling to me.
Re:Meson 1.0 Build System by palata on Saturday December 24, 2022 @08:26PM (#34123099)
I automate it by putting it in a script that I call manually when needed, instead of spending days trying to have cmake run it.

Now it's not a pain for me, and apparently you're on the side of those who find cmake painful. Not sure how unreasonable my way is :)

Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @04:37PM (#34121300)
> For the big majority of projects out there, it seems to me that it is perfectly fine to handle that manually, because projects are not that big. Do we agree on that?

Let's also agree that every time you change a .cpp file you compile it, and every time you change a .h file you compile all the affected .cpp files.

No we don't agree on that. That is the entire point of having a build system.

Even proto files you gave as an example, may import other proto files. When you change one of the imported files you have to recompile all proto files that import that file, transitively.

I have, unfortunately, worked with build systems that do not capture all dependencies and systems that capture way too many dependencies. The former resulted in many hours spent debugging non existent problems, and the latter resulted in many hours spent waiting for compilation to finish.

Re:Meson 1.0 Build System by aseipp on Friday December 23, 2022 @09:36PM (#34113285)
No, in most build systems it often can't be expressed precisely. It can be done in a way that will mostly work or be over approximated, though. So having some amount of abstraction capability is really important. Especially when you have a build pipeline involving something like "compile program G -> run G -> compile output of G", the output of G can often have many dependencies that are not statically known, or you have to repeat the logic of the generator program G inside the build system to declare them correctly, which is often error prone. So you normally just either have some incorrect logic where the build will fail -- because G started outputting code that needed "foobar.h" and someone forgot to update the dependency list to include "foobar.h", so now dependencies aren't tracked -- or some wildly conservative dependency graph that rebuilds things way more often than necessary, "the output of G depends on every header file possible."

Most people either work on programs small enough where having a conservative approximation of the dependency graph is "fine" and excessive rebuilds are OK, or structure the project with a little repetition between the between the build time dependencies and the code. Or, they just let the module system in their language of language choice figure it out. These are all completely OK. But it's very expensive to do conservative recompilation and very error prone to do manual repetition in larger projects, and gets very complex in multi-language setups.

Re:Meson 1.0 Build System by inkyoto on Saturday December 24, 2022 @12:42AM (#34114334)
> […] and excessive rebuilds are OK.

Where does the presupposition of just some build tools, but not the others (if I am reading it correctly), resulting in excessive rebuilds come from?

Apart from building a dependencies DAG for a project, the next fundamental responsibility of a build tool is to watch for / detect changed graph nodes and apply relevant build rules (expressed as graph edges) to rebuild only the necessary parts of the graph by traversing the path upward (if the apex node is up at the top). Naturally, if it is a dependency at the very bottom of a very leafy graph that has changed and upstream nodes have a dependency on it, it will trigger a large (of a very large) rebuild of the entire dependency path. But that is true of all build systems, not just select ones, and that is what we need buid tools for.

The problem is that many people either can't or don't express dependency rules correctly or dependencies generate build byproducts that are to be dynamically injected into the DAG, and they not properly accounted for in the build rules, and that results in an incorrectly built DAG at the build system run time resulting in excessive rebuilds.

In the ideal world, dependency management is relegated to a module system of a programming language. But since long serving incumbents cough cough C, C++ and Java (not until Java 9 anyway) do not have a module system, people have created a kludge (make) and later multiple kludges (autoconf/automake, cmake, maven/ivy etc) to bandaid the problem.

However a generic build tool can't account for a permutation of all possible scenarios, all possible programming languages, linkers, pre- and post-processors and whatever else needs to be done to produce a build artefact and excel at it in every possible use case. Some do it better than the others in very specific scenarios but do poorly in some other or fringe use cases. Then there are personal preferences. Therefore, the discourse over which build system is better than the other(s) will never end.

Re:Meson 1.0 Build System by charcircuit on Saturday December 24, 2022 @05:11AM (#34115527)
Modules don't expose file level dependencies which is what you want as a build system.
Re:Meson 1.0 Build System by inkyoto on Saturday December 24, 2022 @09:06AM (#34116899)
Yes, they do, and I can't think of an operating system (other than OS/400) where a module is not a single file or a collection of files. Could you please reference an implementation where it is done otherwise?

The module system is tasked with figuring out dependencies (i.e. files comprising a module), which is true for all programming languages starting from Standard ML through to Rust and anything in between.

Re:Meson 1.0 Build System by palata on Saturday December 24, 2022 @06:38AM (#34115965)
I did not mean to sound like CMake is perfect for all projects. In some specific situations (typically huge projects), I can totally understand that it becomes a pain. That's why big companies like Google tend to have their own tool.

Now it really feels that most people complaining about CMake and using Meson instead are not in this situation. And if they were, it would seem more pragmatic to me to say "CMake is fine for small projects, but at some size we needed something else" than "Meson is obviously, definitely orders of magnitude better because it took me less time learning how to compile my 20-files project".

Re:Meson 1.0 Build System by aseipp on Saturday December 24, 2022 @02:17PM (#34120131)
Okay, except your whole reply to OP started by replying someone who is in this situation and does have this exact problem. Then you go and ask why their solution can't be simple, and when it's explained why, you say, "well, most people don't need this?" ?????? You're talking to people who do need this!

And this isn't something that ultra-massive billion line projects singularly suffer from. You can easily suffer from all these issues and more in small and big software projects alike. I've suffered from these problems plenty in FOSS and commercial software that were small and large. "Computer program that generates another computer program" is actually a pretty normal and fine way to build software. But when compilation is expensive, these things matter, and you can't keep making excuses about it.

A build system should accurately track dependencies and rebuild things. It's that simple. You can say that running two commands is OK or whatever, but don't try to imply that people who want that criteria -- accurate deps and rebuilding -- aren't trying to solve problems they actually have, and then ditch the example when it becomes inconvenient to you that they are.

Re:Meson 1.0 Build System by palata on Saturday December 24, 2022 @08:22PM (#34123073)
Well my issue is that I have used CMake quite a lot, with generators, in non-trivial projects (both open source and proprietary), and I have always been fine with autotools, cmake and meson.

I don't try to have cmake do too much, e.g. I don't run the generation (or python scripts or custom functions) from cmake, because it quickly becomes a mess and running the steps manually / in a script is usually easy.

I haven't seen a project yet where the cmake config is not a mess (because let's be honest, most CMakeLists out there are pretty bad) and where this is clearly a limitation of cmake (and not of meson).

Happy to get an actual open source example where that's clearly the case, though.

Re:Meson 1.0 Build System by jcelerier on Friday December 23, 2022 @10:13PM (#34113529)
> One thing that I find make CMake configurations completely unreadable is to try to have it do everything, coffee included.

and to me, build procedures that aren't completely implemented in cmake but require to install who knows what shell / platform / language / etc... in addition are the bane of my life as they make everything harder and more brittle and less cross-platform.

Re:Meson 1.0 Build System by rcxdude on Saturday December 24, 2022 @04:34AM (#34115347)
The build system should encapsulate everything needed to build the software, ideally with a single standard command. If your build system requires you to manually run build steps before or after it, your build system is broken (cmake unfortunately also often falls down in this regard, especially for cross-compiled projects).
Re:Meson 1.0 Build System by palata on Saturday December 24, 2022 @06:04AM (#34115799)
So it's fine for you to have to manually run a sequence of steps (start your computer, open a terminal, navigate to the project, run the build, run the program), but it's unbearable if the build is split in a couple commands?

I don't get it. Also shell scripts are pretty good at running a couple commands, if really that's important.

Re:Meson 1.0 Build System by dvdkon on Saturday December 24, 2022 @06:27AM (#34115910)
If you start using shell scripts, you are throwing away the whole advantage of build systems, which is only rebuilding what is necessary. Using multiple manual build phases (even if you have a shell script calling them all) also increases contributor friction and is terrible for FLOSS.
Re:Meson 1.0 Build System by palata on Saturday December 24, 2022 @08:34PM (#34123150)
Well, I run the scripts only when necessary. If I change an IDL file, I run the generation script. If I change a cpp file, I run the cmake build only.

I actually have maintained FLOSS projects, and in my experience, throwing everything in one cmake project makes it harder for contributors to understand. Because they only ever use one command, and the day something fails they have to dig into it to understand what that custom command does. On the contrary, if they need to run the generation script first, then they know that there is a generation step involved, and suddenly the whole thing feels less like magic.

So that is my opinion: if your FLOSS project uses a big custom build system (because you customized your cmake to do all sorts of custom things), then probably I will not contribute. Because I have other things to do than learning your custom integration of standards steps I actually would already know.

Re:Meson 1.0 Build System by rcxdude on Sunday December 25, 2022 @07:18AM (#34126597)
Why do you feel it's better to do the build system's job manually? I've seen beginners interact with such a system and it's a recipe for even more confusion.
Re:Meson 1.0 Build System by palata on Sunday December 25, 2022 @06:58PM (#34132918)
> Why do you feel it's better to do the build system's job manually?

What's better is what is easier to understand and maintain. If your whole project is made of one file and builds with one simple call to a compiler, IMO it's fine to just have this line in the README, no need for a custom_target in cmake.

Now the thing is that often, I see that people put a lot of importance into the "it should be easy to get started for beginners who don't know the tools", and everything is glued together using some custom functions. The goal being that a beginner can essentially run "make" (yep, I've seen Makefiles wrapping CMakeLists for that) and the project will build (sometimes it will even install dependencies on the system, that's crazy). But then if the beginner wants to understand what's happening under the hood, well... it's a big pile of custom glue.

Now if, instead of that, you document the tools you are using in your README, and use the tools in a standard way, then people who know the tools will get started quickly ("oh, it's protoc then cmake, I know exactly how to use that"). And yeah, beginners will need to learn how to run those tools. But that's fine IMO, because if you use the tools the standard way (without custom functions), then the beginner can just look for the official docs.

Standard is better than custom.

Re:Meson 1.0 Build System by rcxdude on Sunday December 25, 2022 @07:16AM (#34126581)
I don't want to have to find out about and remember a bunch of different commands, and I especially don't want the risk that I get into an inconsistent state by forgetting a step when something changes.
Re:Meson 1.0 Build System by junon on Friday December 23, 2022 @07:56PM (#34112575)
Don't bother arguing with Meson contributors. The typical response to realistic use cases like this is to gaslight you into thinking you're doing something wrong.
Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @06:17AM (#34115860)
LOL

There is definitely a religious feeling to some of their arguments.

Re:Meson 1.0 Build System by bonzini on Saturday December 24, 2022 @07:02AM (#34116077)
Not necessarily wrong, but you might have to try doing things differently and it might work or not.

My experience with QEMU's transition to meson is that it was clear from the beginning that Meson would not be able to do everything we needed, but it could do most of it. We live with the limitations and the outcome was extremely positive:

* 4000 less lines of build system gunk

* A lot fewer requests for help

* Several new features that could be implemented much more easily

* A transition that was

Of course there is still complexity, QEMU has 11000 lines of build system code. It was not easy and it involved several contributions to meson itself (that everybody now benefits from, for example Postgres) but it was totally worth it.

Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @12:48PM (#34119148)
I'm sure it is better than the previous system, otherwise QEMU would not be migrated to Meson.

But, you're describing a 11,000 line program without the most basic abstraction mechanism known to man - user defined functions.

What are the benefits of not having user defined functions? none, zilch, nada.

The Meson team is grasping at straws trying to claim that having user-defined functions would make it Turing-complete. Without recursion, it won't.

Try to imagine how you would structure those 11,000 lines of code if user-defined functions were available? Would you really write it without extensive use of functions?

Re:Meson 1.0 Build System by bonzini on Saturday December 24, 2022 @04:12PM (#34121136)
It's 11000 lines (for 2.5 million lines of code or so) but it's really a long list of things to do. Functions to look for, libraries to use, configuration symbols to associate with a specific file.

The only somewhat repetitive idiom is

foo = dependency('name', method: 'pkg-config', required: get_option('opt').disable_auto_if(condition))

to find a library, but otherwise there's very very little repetition of the kind that can be fixed with user-defined functions.

That's not surprising though. Because the user input is very constrained, I find it more natural to think of the build as "first prepare all these things, then use them to prepare all these other things" and so on, not as a complex program. As a very simple example, instead of abstracting "find flags for the C/C++ compiler" in a function and calling it twice, I prepare an array of languages and iterate on it.

It seems limiting but really it's just different and this kind of data flow mindset makes it very clear what is happening when (e.g. what files or arrays are read and which are written), and it also does not need a lot of abstractions (functions).

What it needs is decent data structures, and Meson's DSL is superior to CMake in that respect. So basically the code you write matches the tools you have.

Now I am not saying that Meson is perfect. But the functions thing is kind of a red herring. My #1 complaint is lack of include files (apart from going down to a subdirectory), not functions. #2 is messy handling of targets that generate files in subdirectories and #3 is a silly handling of some options.

The important thing is that none of these things, and not even functions for that matter, are baked in the language. There is room for fixing them in the future, because they are missing features or bugs rather than fundamental mismatches.

To be honest I believe that Meson should have used Starlark as the language, but it did not exist 10 years ago and anyway there is no Python implementation of it.

Re:Meson 1.0 Build System by germandiago on Saturday December 24, 2022 @06:29AM (#34115918)
AFAIK this can be done with a mix of custom targets and adding the result to the input of the shared lib in some way.

What thing you cannot do? I have used capnproto, generated c++ files and passed the result to the input of targets perfectly ok.

Re:Meson 1.0 Build System by oldgradstudent on Saturday December 24, 2022 @12:51PM (#34119189)
Of course O can do that. The point is that I want to do it many times over, in different places in my code.

What I want to do is to spend the time and effort in understanding and debugging the process, and then encapsulating it in a single function call. Then completely forget about the details and use that single function call.

What is the advantage in not having user-defined functions? None, zilch, nada.