Highly recommend reading https://ocw.mit.edu/courses/electrical-engineering-and-compu... for some theory on parallel execution for graphs, if your interested in things like this.
Also be sure to have a look at tup, which operates vastly more efficient by simply walking the DAG in the opposite direction:
That is, instead of looking what you want to build and checking all timestamps of dependency files, it can use e.g. inotify, then know exactly which files changed, and rebuilds everything that depends on those files.
Moreover, it performs the modification check only once, at the beginning. During the build, it doesn't need to re-check everything, because it already knows which files were recreated.
I have foo.c, bar.c and a rule like:
: foreach foo.c bar.c |> ^o^ gcc -c %f -o %o |> %B.o {objs}
: {objs} |> gcc %f -o %o |> baz
The ^o^ part tells it to not trigger the next rules if there's been no change to the output. So if you just change your source files to use an updated license, reformatted it for clarity, etc., then nothing else will happen.I used this with some of my literate code projects where I had tup running org-tangle for me (via an emacs script). If I'd only updated the documentation, and the code hadn't changed, nothing else would build. If I'd only changed unimportant parts of the code that generated the same object files, no new binary or library would be built.
I believe that tup does have some features in that direction, but I may be mistaken.
Maybe someone who used tup more recently can pitch in
Any CI I've seen starts from a fresh repo checkout and rebuilds everything every time, so it's not an issue with practice.
OTOH, probably all my projects were small enough for it to not hurt when the CI builds everything from scratch every time. I might look at this from another angle if the things I worked on were mega-LOC C++ programs, and not kilo-LOC Go programs.
Honestly, if I were to modify make for some custom behavior, I would look for another tool¹ or create yet another builder tool. Because although I already know most of the syntax, point #2 is an incredibly big problem, the GP missed point #4 in that make does not manage external dependencies², and the syntax is not bad only for learning, but for using too.
1 - Almost everybody seems to agree with me on this. The only problem is that we have all migrated to language constrained tools. This is a problem worth fixing.
2 - Wether it is installing them or simply testing if they are there, make can't do any. Hello autotools, with it syntax set to extreme cumbersomeness.
Make is about "making files." Or to be a little more semantically specific, Make is about "processes that transform files to make new files based upon their dependencies". It really doesn't matter that your file is a C source file, or some unpreprocessed CSS, or a template, or an AMI. To your AMI example, if you specify a dependency (or dependency chain, DAG) as a time-stamped (or set of) file, you can get make to rebuild the AME for you along with any other supporting or intermediate files.
IMO, the suckless guys are masters at writing deceptively complex but highly readable & concise Makefiles. Here's plan9 workalike utility library [0], a util-linux/busybox package of binaries [1], a window manager [2], a terminal emulator [3], and webkit2 based web browser [4]. I highly recommend you study them if you're looking to up your Make game.
[0] http://git.suckless.org/9base/tree/Makefile
[1] http://git.suckless.org/sbase/tree/Makefile
[2] http://git.suckless.org/dwm/tree/Makefile
I'm not saying you can't use make (we did use make before we switched to walk), it's just more painful for non C/C++ build systems. All we really want from a build systems, is a generic tool for describing, and walking, a graph of dependencies.
If you're careful (and even if you're not), loops in your dependency graph are usually a non-issue. And if you use a single `Makefile` it'll detect the circular dependency and try to ignore it.
Let's take an example processing Sass CSS files. You just need 2 folders in your project, `css/` and `scss/`, and a `Makefile` to process all your Sass files into CSS.
# Locate our source assets, save a list
SRC = $(shell ls scss/*.scss 2>/dev/null)
# Specify a filename transformation from scss to css and convert the SRC list
OUT = $(SRC:.scss=.css)
# Define rule of how a scss to css transformation is supposed to happen
css/%.css : scss/%.scss
sass $<:$@
# Make the `all` target depend on the OUT list
all : $(OUT)
It takes just 5 lines to teach `make` how process Sass files. This would process any `.scss` file you dropped in `scss/` and save it in `css/`.If you use @import in your scss file the css will not be rebuilt if only the imported file is modified.
So no. It does not take 5 lines to teach make how to build Sass. To solve this you either need to teach make about sass @import, or teach sass about make and let it generate makefiles (like gcc -MMd). Or simply just use sass --watch for incremental builds and take the recompile hit if you need to restart it for whatever reason.
(While I'm at it...Another thing missing from that makefile is source maps (.css.map) generated by the sass compiler. It's not only one css file being generated from each sass file. That will complicate the rules even further)
To fix the `.css.map` issue is simple. You can't use the pattern you've seen for yacc/bison, though. The `$@` var would have both files in it.
%.tab.c %.tab.h: %.y
bison -d $<
Instead, simply adjust the pattern to rule to be the functional equivalent: css/%.css : scss/%.scss
sass $<:$@
css/%.css.map : scss/%.scss
@touch $@ css/%.css : $(SRC) scss/%.scss
sass $<:$@
css/%.css.map : $(SRC) scss/%.scss
@touch $@And yes, by that logic, `SRC = $(shell ls scss/*.scss)` is dumb. That is not lost upon me.
Or teach a third-party tool to output a list of @imported files for a given Sass file. Then this list can be used as that Sass file's dependencies.
By the way, I just noticed that compilation times are another argument for microservices.
I meant "in practice", of course...
Note that "recursive make is harmful" does not argue against multiple Makefiles!
There's nothing wrong with using multiple Makefiles per-se, as long as they include rather than call each other. In other words, the article just says that Makefiles should use sub-Makefiles via "include" rather than executing those through separate (recursive) call to Make.
However, I agree that composition of Makefiles is still a pain, given that the included Makefile must be aware that it is executed from another (parent/grandparent) directory.
This is my #1 gripe with Make, and many other build systems as well. There are so many flaws in the timestamp approach. Most are easily fixed with cryptographic hashes.
I like the OCaml build system OPAM for that matter, it internally just stores checksum. I believe it also uses timestamps to speed things up, but only for equality comparison (not for older/newer comparisons which may easily lead to wrong result).
But that's kinda silly.
I might argue in favor of (fast) cryptographic hash algorithms in general since they're fairly well understood / implemented / hardware accelerated / tend to have extremely "balanced" random output thus less likely to accidentally conflict... but that's about all I can think of.
Multiple Makefiles works fairly simply with the include directive.
The thing I've found painful before, with C projects, is getting Make to recognise that it needs to rebuild when a header has changed. There are various ways around this (makedepend etc) but they've all been quite painful to set up and not quite perfect.
That said, with modern machines that have NVMe storage and massively fast processors, a complete rebuild is seldom a big time cost
http://make.mad-scientist.net/papers/advanced-auto-dependenc...
You already tried that? I do not find it painful to set up.
The syntax is a tad hairy though,and I'd want to adapt it - I tend to avoid compiling individual C files to objects these days, due to WHOPR optimisation.
That said, I think incremental builds are important for most use cases.
On the other hand, computers are fast enough that serial full builds indeed work in some cases.
And btw there is no way to use make without shell, but you can use shell without make.
Sure, I can write these as functions in bash, call once for each source file, check return codes etc etc, but I find expressing dependencies faster than writing anything like robust code in shell, and make deals with failed steps by stopping.
Are Android and LEDE/OpenWrt big enough?
> 3. Let's be honest, nobody really wants to learn Makefile syntax.
That's probably very subjective, I find JavaScript, C++, PHP or Perl much worse :-)
Anyway, for the past years I'm cheating a lot and using CMake to get complex Makefiles almost for free.
Oh definitely. I actually like Makefile's, but in my experience, more teams have a deep familiarity with some programming language, than with Makefile syntax. I haven't met very many people who have read the GNU make manual, and know all the idiosyncracies around Makefile syntax.
It feels much more like in those IDEs where you just drag all the source files in and you're done.
I even prefer editing visual Studio project files by hand to many makefiles out there..
Doesn't mean that it's not hard. :) And if IIRC google are trying to move android build to blueprint because make is too complicated.
target/linux/ar71xx/image/Makefile: CMDLINE = $$(if $$(BOARDNAME),board=$$(BOARDNAME)) $$(if $$(MTDPARTS),mtdparts=$$(MTDPARTS)) $$(if $$(CONSOLE),console=$$(CONSOLE))
If BOARDNAME is defined, add board=$(BOARDNAME) to the string
If MTDPARTS is defined, add mtdparts=$(MTDPARTS) to the string
If CONSOLE is defined, add console=$(CONSOLE) to the string
Pretty simple, if a little like the ternary operator in C.You must have seen more complex clauses than that in shell scripts and all sorts of places.
`unzip -D` is your friend.
My makefiles are usually just a way to record a data pipeline. Get these files, shove them through these scripts here and those programs there. Launch a web server to show the output.
a) Mtimes-as-change-detection is fundamentally broken given the reality of networked file systems and... physics. (Minor problem, but extremely annoying to work around in practice.)
b) Nobody can actually really understand all the interdependencies between all the code in their system(s!), and yet Makefiles expect you to specify all of that explicitly?!? Yes, you could technically specify that -- and you'll want to -- but you won't, because you don't know and don't have the time.
c) Make support for builds that change the structure of the build during the build is abysmal. E.g. after "processing foo.xml we now have more files than we had before! What are you going to do?". Well, in Make it's some sort of custom solution with ".d" files and "gcc -M" or whatever. This is utterly broken in that it pushes all the complexity onto the user. So, yes, an elegant model, but it doesn't actually solve the problem. If you want to see a better solution see the "Shake" paper.
I haven't found that to be the case. Once I show people how simple it is they realise make is somewhat approachable, similar to OP's article.
Your README’s example show `parse.h` as output from `Walkfile deps parse.o`, I think that is a mistake.
As for your build system (and comment), I have some questions:
1. How do you achieve using a deterministic hash as condition (and aren’t all hash functions deterministic)?
2. Why would you not be able to use mtime as a dependency? The only case I have run into is when the build depend on data from remote machines, but in that case, I think the proper solution is to have an initial “configure” step where you retrieve the data your build depend on and/or a build rule to update this data.
3. Does your build system execute the Walkfile for every node in the graph on each walk? Because that sounds like a quite noticeable overhead for larger projects.
4. Am I right in thinking that the primary advantage with your system, over make, is that a shell command is executed to obtain a target’s dependencies?
2. Mainly because doesn't translate across machines; it only applies to your machine. If someone checks out the repo on their machine, mtime is different. As soon as you move a build system to CI, you need something better than mtime, like content adressable hashes, if you intend to cash targets.
3. It executes the Walkfile twice for each target in the graph; once to determine the targets deps, and once to execute the target. This definitely hasn't been a bottle neck for anything I've used walk(1) with so far.
4. Correct! But even more so, replace "shell script" with "executable". The Walkfile can be written in any language you want, as long as it has the executable bit set.
This is one of those things that "distcc" and "ccache" have dealt with effectively - anyone trying to build big C++ projects would be well served by looking at those tools.
As for #1, what I do not understand is how a `Walkfile` allows me to use a content hash change to trigger a rebuild.
Your documentation says that a file list should be returned for `deps`, so how does the `Walkfile` communicate that e.g. `main.o` should be updated if `sha1sum main.c` is different than on last invocation?
That's a trivial example using mtime. You could replace it with a hash function if you wanted.
Only if each Makefile is treated as a separate rule set processed with a separate make invocation.
> Read the article about why recursive make is harmful
That (now) venerable, old paper in fact shows how to break up into multiple makefiles (called "module.mk" in its examples) which are included into one graph.
(It's possible to actually have this file be called Makefile. Not only that, but it's possible to have it so you can actually type "make" in any directory of the project where there is a Makefile, and it will correctly load and execute the rules relative to the root of the tree.)
Make's syntax is quite simple. It's a bunch of variables one has to memorise, and man page is a command away. And any decent editor would know to insert literal tabs.
> Make is _really_ hard to use to try to compose a large build system from small re-usable steps.
I have no experience myself, but Linux's build system is a bunch of homebrew makefiles, all the BSDs and their ports trees build with bmake. These are enough positive examples for me to think that Make is good for big systems.
I'll just leave this here:
https://github.com/sapcc/limes/blob/62e07b430e2019a6c1891443... (then used in line 42)
>walk is currently not considered stable, and there may be breaking changes before the 1.0 release.
:)
It also needs go, which is pretty non-lightweight dependency on windows, I guess. (And personally, I don't find these dir-hierarchy connection and bash's "case $target in a) ... b) ..." any friendlier at all, though make has some quirks with variable interpolation.)
I'm not to argue to use make for very complex situations, but usual src->intermediate->executable and generate this from that of any size and count is a perfect task for make. Makefiles are unsuitable not for big projects, but for complicated build systems, where it's not enough to just do them steps in correct order. If your build system is complicated, it should be worth it at least. Otherwise use make.
Fix: typos/grammar
If you don't like bash syntax, you can use Python, or Ruby, or Perl, etc. Any executable can serve as a Walkfile.
It's a completely fair point that make is installed on pretty much every machine by default, which is why we won't see it going away anytime soon (nor should it, it's still good).
1. Using make in a CI system doesn't really work, because of the way it handles conditional building based on mtime. Sometimes you just don't want the condition to be based on mtime, but rather a deterministic hash, or something else entirely.
2. Make is _really_ hard to use to try to compose a large build system from small re-usable steps. If you try to break it up into multiple Makefiles, you lose all of the benefits of a single connected graph. Read the article about why recursive make is harmful: http://aegis.sourceforge.net/auug97.pdf
3. Let's be honest, nobody really wants to learn Makefile syntax.
As a shameless plug, I built a tool similar to Make and redo, but just allows you to describe everything as a set of executables. It still builds a DAG of the dependencies, and allows you to compose massive build systems from smaller components: https://github.com/ejholmes/walk. You can use this to build anything your heart desires, as long as you can describe it as a graph of dependencies.