Like many others, I too rediscovered the power of pure GNU Make years ago. Turns out it has some rather Lisp-like metaprogramming capabilities. It's extremely hard to do even basic processing but with enough effort some cool things can happen. I'm somewhat proud of my implementation of a simple version of the find utility as a pure GNU Make recursive function.

This rediscovery essentially derailed one of my side projects as I started writing a pure GNU Make build system and function library like this one:

https://gmsl.sourceforge.io/

https://www.cmcrossroads.com/article/gnu-make-standard-libra...

I even asked this author on Twitter and Stack Overflow about its implementation.

The end result was reinvention of code generation tools because that's what metaprogramming is. I didn't understand what autotools was doing before I tried doing it myself. My objective was to eliminate makefile regeneration whenever project structure changed, I ended up regenerating code 100% of the time and evaluating the resulting code inside the makefile itself. If we must metaprogram, it's better to do it in a proper programming language. At least it didn't have any dependencies other than GNU Make.

Hello! That’s me. Happy to answer further questions…

Hello! Huge fan of your work and I've also read your book. Are you still developing GMSL? I wonder if you're open to contributions.

My find function has been the single most useful thing to come out of my side project. The first thing I do in my makefiles is get a list of all source files and it's always been annoying because I like organizing my project in trees which requires recursive file system walks. I managed to write a pure GNU Make function that simplifies that to:

  sources := $(call find,src,file?)

The find function recursively walks the src directory and returns a list of all paths satisfying the file? predicate function.

I think it would be a nice addition to GMSL. In my project, I doubt it will ever help anyone.

Yes, I still work on GMSL. It's pretty stable and I mostly just fix incompatibilities or bugs that people report. However, I'm happy to look at additions.

One of these days I'll get it off SourceForge...

Is your find function a bit like my rwildcard? https://news.ycombinator.com/item?id=23270235

That's awesome. I didn't know about rwildcard until now. Is it part of GMSL? I searched for rwildcard on gmsl.sourceforge.io but didn't find it.

I think my function is needlessly complicated compared to rwildcard. Here's my code:

https://github.com/matheusmoreira/liblinux/blob/modular-buil...

https://github.com/matheusmoreira/liblinux/blob/modular-buil...

The file? and directory? functions were inspired by GMSL. I think I learned how to check whether a path is a directory from one of your posts.

I wrote a general recursion function. It takes a function to apply to lists and a function to compute whether an element is a base case.

The recursive file system traversal function applies a directory globbing function to the list of paths and has file? as base case. It returns all files without filtering.

The find function recursively traverses the file system while filtering out all items not matching the given predicate function. It was my intention to provide predicates like c_file? and header_file? but I stopped developing that project before that happened. Users can easily write their own predicates too.

I think rwildcard is probably simpler and more efficient! If my code helps in any way though, feel free to use it.

"Like many others, I too rediscovered the power of pure GNU Make" ... wait a sec, GNU make is not "pure!" I like it but I recognize that it has features that a more portable make would not have.

Perhaps pure was not the right word. By pure GNU Make I meant writing makefiles myself without higher level makefile generation tools.

Honestly, I never quite understood the point of generating the makefiles: if you have a tool that knows where the inputs are, where the outputs should be, and tracks the dependencies changing from build to build (something that "make" famously struggles to do), then surely that tool can also run whatever compilers/linkers/etc. are needed?

I guess you can commit the generated makefile into your repo so that those who want to build your project from sources don't need your custom build tool installed, only make...

Historically, Makefiles generation is merely a cache of platform introspection on systems that were orders of magnitudes slower than what we have now. I suspect that any other approach based on Makefile only would have implied higher memory consumption and/or implied slow rotating disk I/O.

Actually on my 64-thread Threadripper that cache is more important than ever as the make step parallelizes nicely but the configure step usually is 100% serialized.

Take a look at FreeBSD. Here’s what a typical Makefile looks like:

# $FreeBSD$

PACKAGE=runtime PROG= setfacl SRCS= file.c mask.c merge.c remove.c setfacl.c util.c

.include <bsd.prog.mk>

So, yeah, at least for code in C you can have something maintainable in make(1), it’s just you need to structure your project in a sensible way.

Does it support incremental builds? How does it detect include dependencies? Is it portable?

1. Sure (if you didn’t need incremental builds, why bother with make at all? Just use sh.)

2. It doesn’t, it’s programmers job to make an effort to append a file name to a single line in a single file.

3. It’s portable in the same way autotools are portable, which means you can make it work with relative ease.

Most builds are divided into two phases. The first phase "do we have libfoo installed, where is doxygen, are tests enabled" is run infrequently. The second phase "recompile everything that needs to be recompiled" is run over and over again. The second phase is more or less a solved problem now given that Ninja exists. The first phase is where a lot of people have different ideas and tastes, so we get cmake, meson, autotools, etc.

You can certainly do everything in one system. In practice this seems to lead to a build process that is much slower than it needs to be, see for instance Scons.

Or see cargo, go build, dune, etc.

These are all tools that specialise in building only a particular kind of project. They are useful, but aren't really in the same category of general build tools like Ninja and Make.

There are many nuances. How to control parallelism, how to present output, checking freshness, etc. It is much easier to generate Ninja files and let Ninja do all the boring work.

What are the dependencies of Ninja? (I seem to remember it being a Python tool?)

On macOS at least, Ninja is just a 362 kByte native executable without dependencies except dynamically linking with the system-provided libc++ and libSystem DLLs (you're probably thinking of Meson?)

Ah, that might be it! Thanks!

Do you want to hand parallelization in your little custom tool?

Well, cargo has it. So does go build. And dune. And MSBuild. And... basically, every language-specific (and non-specific) build tool out there handles it, so yes? Sure, it's not exactly according to the UNIX philosophy but judging by other comments in this thread, if one wants a low-level parallelizeable build tool, they should use Ninja instead of Make anyhow.

> …but, make doesn’t scale;

What about these projects?

https://github.com/redis/redis/blob/unstable/src/Makefile

https://github.com/torvalds/linux/blob/master/Makefile

Yes, I can also write a big Makefile to hand compile my project; I can also compile it using shell script that explicitly executes the build steps.

…but I dont, not because it’s impossible to do but because the effort in maintaining those is high.

Come on, are you serious? Hand crafted makefiles? Have you actually tried to maintain one of those? That redis makefile has a explicitly coded set of object files. Can you imagine trying to maintain that for opencv?

I’m just too damn lazy to waste my time maintaining that kind of stuff; it’s a pointless, thankless task… but hey, I’ll give it to you; there are people out there who prefer to do it that way.

I posit, they spend more time maintaining their build system then they would prefer, but they keep make for ideological reasons, not practical ones.

Are you aware that the first step in building linux kernel is literally bootstrapping a custom build system that just happens to generate makefiles (if not just "mostly make compatible" files - I got crosseyed recently trying to find if kbuild didn't try to run them as well)

Linux kernel provides "makefile" as interface for you, but it's mostly compiled from a custom KBuild system.

AFAIK, the makefile fragment generated by kconfig is nothing more than a set of CONFIG_FOO=y (or =m) statements; everything else is done directly in the makefiles, which have declarations similar to "obj-$(CONFIG_FOO) += foo.o" which become obj-y or obj-m (and so on) depending on the value set by that makefile fragment generated by kconfig. The set of object files to build can then be found in "$(obj-y)" (and so on), which is used as a dependency in a makefile rule to link everything together. The same kconfig also creates a C header file with #define rules for these same CONFIG_ symbols, which is passed to the compiler through a "-include" argument. It's not "mostly compiled from a custom KBuild system", IIRC kbuild is the name given to the whole system, including the makefile rules and macros which do the actual compilation.

Decided to go into the source again, and yes, it's mostly implemented in Makefiles, but they are wildly different from the kind of makefile discussed in the article, with considerable amount of machinery in macros and external C programs (whole "features" thing) to the point that individual makefiles in subdird probably wouldn't parse right if not executed through kbuild wrapper

You offer plenty of criticism for using and/or generating makefiles, but I can't find any place in the discussion where you suggest an alternative.

So what is your solution for building large projects?