That would be the "bootstrapping" process. Nearly a half-century ago I took a compiler lab class where we were given a working, but slightly lame, compiler, and were tasked with adding new, less lame, language features by bootstrapping. That is: 1) implement new feature without using the new feature, 2) using the compiler that results from step 1, re-implement the feature using the new feature, and compile again. 3) Repeat with more features until end of semester for best grade.
Oh, and to the OP, well done!
In all seriousness, the bootstrap process is fascinating to me. At some point, it did all start with switches and manually loading commands directly into memory. And over time, we’ve slowly kept this thing going and growing. Also… I’m old enough to have seen a Altair with toggle switches, but I’m not old enough to have had to toggle in a boot loader on one. :)
Taking this process to the extreme, no one could ever bootstrap anything because eventually you're bootstrapping mineral extraction.
https://bootstrappable.org/ https://lwn.net/Articles/983340/
https://bootstrappable.org/ https://lwn.net/Articles/983340/
- Breadboarding some CPU components
- Breadboarding a very, very minimal CPU. More like a simple Adding + Multiplication machine with 4 registers or so, with a few premade components.
- Eventually moving up to microcoding a simulated CPU
- Eventually writing binary code to control the microcoded CPU
- At that point I kinda cheated and wrote my own assembler because I got sick of checking so many bits.
- This project then stopped at an assembly level.
- But then we implemented our own ML-variant interpreter and later on ML-variant compiler in OCaml
- And later on we had that ML-Compiler compile itself and extended it from there.
Pick your poison where to start. Just be aware that breadboarding stuff becomes very... messy very quickly[1]. And I do include Hardware in this, because you needed simpler CPUs to design more complex CPUs.
In practice, you want to pick the best combination of a familiar language that is as high level as it can be. But back in the day, that would've been assembly, binary. In the case of C, it went from BCPL, in which a compiler for B was written, in which a compiler for New B (NB) was written, which then turned into C.
My guess is that some people would rather write Python code than dig into Make’s documentation. That said, ChatGPT is excellent at creating valid make files.
The thing about Makefiles is that simples ones at least are really easy to write, and read. Much simpler and quicker than a cumbersome python script, that will most likely do less with much more boilerplate code (e.g. dependencies), and be much harder to read.
Of course, you may hit a point where you stretch your Makefile so much beyond its common capabilities that that no longer becomes true, but in my experience that point is pretty far away.
Less even, here is an attempt:
foo.o: foo.c
clang -c -o foo.o foo.c
fooexec: foo.o bar.o
clang -o fooexec foo.o bar.o
Now type this into the shell:
make fooexec
But of course all the .c -> .o rules look the same, so instead of writing a new one for every file, you'd use simple wildcards, and now the whole, entire Makefile is just:
%.o: %.c
clang -c -o $@ $<
fooexec: foo.o bar.o
clang -o fooexec foo.o bar.o
It pains me if that is not immediately obvious. And it pains me even more that someone would want to write a boilerplate heavy, complex python program, that rebuilds everything all the time, instead of just learning this very simple concept.
It also has “phony” targets that aren’t files. The canonical example of this is “clean” which basically just does something like rm -rf *.o fooexec
This is what the JavaScript tooling people have been chasing for the past 15 years and can’t seem to grasp. Someone actually did build a make-based build system I believe but it never got popular because it wasn’t written in JavaScript.
A nitpick, but I don't think that should really be called caching, although it kind of is, in a looser sense. It's just comparing the file timestamps of the .o and .c files, and building the .o from the .c, if the .c is newer, or if the .o doesn't exist.
So, either you need to manually keep references to which .h files you include in your Makefiles up to date, or start worrying about M / MM / MG / MP, and of course you'd like those to be re-run when you change your files, and suddenly your Makefile is an awful lot less simple.
This is the main reason I stopped teaching Makefiles in intro to C courses -- students often lost entire afternoons to forgetting this stuff, and failing to realise their mistake. It really shouldn't be any person's job to keep this up to date, when the computer knows it.
That is not necessarily the most useful. Especially as your project gets larger, you probably will not want to specify every header file as a manual dependency, or alternatively rebuild everything every time you change a header file. (Though for me personally, either approach often works for surprisingly long.)
Then you can do things like letting the C preprocessor create the dependencies that get included in your Makefile (yes, common C compilers can create Makefile fragments). Or do something more complicated. Or switch to another build system, if your project really outgrew, or really mismatches, make.
But at its core, Makefiles follow a simple concept, are entirely language agnostic, and even with just the simpler concepts they remain useful for a long time.
If you can remember exactly when you need to manually skip the incremental build, that's great for you, but I find Make has enough of these kinds of footguns I don't recommend it to people any more.
As I said, you can let the C compiler generate your header dependencies for you. But that's not enough, because what's with header and libraries that are external for your project? Keep in mind that they will have dependencies, too, and at some point your transitive closure will be really big.
To a lesser degree, what's with a compiler, linker, or any random other part of your toolchain that changes (at least those are supposed to mostly keep a stable interface, which isn't the case at all for external libraries, but in some cases the toolchain can cause the footguns you mentioned).
A lot of the times, you don't need to rebuild everything just because a system header or the toolchain changed, but your build system would have a hard time knowing that. Because at some point, the halting problem even gets in your way (i.e. you can't really reliably detect if a given change in a header file would need rebuilding of a source file, which you would need for "fully incremental" builds). So it's always a trade-off.
Personally, I've fared very well with generated header dependencies (sometimes manually declared ones or none at all for small projects), and many other projects have, too.
YMMV of course, but I don't observe this to be bad. Most people who program C and use make are, I think, aware of what header mismatches can cause, and how to avoid that.
Two reasonable viewpoints, but probably effects how we do our building setups!
I do have to switch between multiple SDK versions very often for example, and I always either nuke the build dir after doing that, or have separate build dirs per SDK in the first place.
Or potentially just specify the path to $CC itself as dependency? Presumably, if the C compiler changes, the timestamp of its executable does too. (Bad if it goes backwards, though.)
Careful, one more step and we'll start recreating nix in Makefiles;)
(Wait, could we? Should we?)
I love it, and I hoped this would become the future of how build-systems are built -- you could automatically parallelise, rebuild, make clean, everything, just by tracking which files are read, and written.
It does add a bit of overhead, but I'd be willing to pay that for simplicity. Unfortunately, it doesn't seem to have caught on.
ccache can cause things to get built twice, because the first run it sees it tries to read a file, then later writes to it, so it knows something different might happen if you run again -- but then again that is just ccache doing it's thing, so it's quick. Yes, distcc wouldn't work, but then again I find distcc super-brittle (so hard to make sure everything has the same compiler versions, etc), I may have used it 20 years ago, but I don't think I ever would now!
And yet, I still don't actually know what `$@` and `$<` actually mean outside of your example. I only can assume that somehow that substitutes in `foo.o` and `foo.c` because you literally put the non-magic versions right below it (and above it a few paragraphs above as well), but I'm honestly not even positive that I'd run `make foo.o` rather than `make foo` with your example, and I feel fairly confident in guessing that this would _not_ compile into a binary called `fooexec` like the target you defined above. I don't have an idea whatsoever what I'd need to put in the target to be able to substitute in something like that because the fact that the first `%` somehow becomes `$@` and the second becomes `$<` doesn't follow any obvious pattern I can use to figure out what the third symbol would be, or even if there is one.
The problem is that "simple" is not the same as "intuitive" or "discoverable". Yes, I could probably put in the effort to go and find out these things, but why would you expect someone to be motivated to do that by someone telling you that it's "trivial" and it "pains" them as they explain things in a way that doesn't actually clarify things in a way that helps me? If you actually want to try to make a strong case that this is something worth learning for people, repeatedly referring to it in ways like "very simple" and "so trivial" and "immediately obvious" is counterproductive.
As an aside, I think it's also a bit of leap to assume that they're "rebuilding everything all the time". It looks to me like the build.py script is only needed a single time to bootstrap from the original compiler to this one, and rebuilding every single C file in the new compiler rather than using the artifacts built with the original one is kind of the whole point. After that's done once, I don't see why the new compiler couldn't be used via the Makefile. If you're complaining about Python build scripts in general rather than this specific one, I don't know why you assume they can't be written to check the timestamps on the files to see if a rebuild is necessary before doing so. That's exactly what `make` does, and hey, it's a very simple concept!
And $< is of course the thing the target depends on. It’s the input, like standard Unix notation for stdin.
Hopefully this helps?
The rest is just... of course you can reimplement make, or a subset of it, in python, python is turing complete after all. But why? make isn't that arcane. I'm sorry it uses the weirdly looking $@ and $< for "target" and "dependency" respectively. Those two are extremely common and you have to type them in a lot of rules, so the authors chose a shorthand. One way or another your python make subset will have to bring in the same concepts they represent, too.
This is trivial in the sense that it's a part of any build system worth its salt. And I can almost guarantee you that OP for example, who wrote their own C compiler, understood it immediately.
The problem is, you typically don't need to touch Makefiles that often once they work (or fulfil a reasonable interpretation of "work"), so unless you're a distro packager or the project(s) you work on are sufficiently complex and large to warrant an entire FTE just for build tooling, chances are high you need to touch that stuff only once every few years, by which time almost everyone has to dig into the documentation yet again... made worse by the fact that Google's search quality has gone down the drain.
My point is that concepts being simple doesn't mean that the choice of how they're represented doesn't matter. When people aren't bothering to learn something you think seems "simple", it's a mistake to immediately assume that it's due to laziness or incompetence rather than first trying to understand their motivations. Maybe they are just lazy or incompetent, but it's also possible that there's legitimate confusion around something that wouldn't occur to you without seeing it the way a beginner sees it, and there's room for improving how things are documented or taught.
There's a pattern of thinking that seems pretty common in our industry where we focus so much on the technical details of a system that the actual human experience of using it gets overlooked. My problem with this line of thinking is that value a tool is only realized when people actually use it. As a hypothetical, imagine that some new technical issue becomes commonplace and two different tools get written to solve the problem; the first tool solves the problem perfectly and as quickly as possible every time but doesn't get used by anyone, and the second tool does only an adequate job of solving the problem and requires a bit of manual additional work from the user to fix things, and that tool gets used by almost everyone. I'd argue that the first tool is not actually successful in any meaningful way because it doesn't actually help anyone with the problem that it was designed to solve.
To be clear, I'm not trying to say that no one uses makefiles or that they don't provide any real world value. I'm trying to say that if you're actually trying to get more people to use makefiles because they solve problems that everyone has better than what they're currently doing to try to solve them, the most effective way would be to look at it from the perspective of the people who aren't already using it and figuring out what's stopping them. When you feel strongly that it's the best way to do things, anybody who isn't doing things that way must see things differently, and the best way to try to change that starts by figuring out what that difference is.
If you don't actually care about trying to convince people and mostly just want to vent, then this advice won't be relevant, but if your goal is actually to try to change the way people do things, the approach you use to explain things is going to matter just as much as your knowledge about the things you're explaining. Just like how a brilliant professor who does groundbreaking research might be terrible at teaching an intro course to undergrads, it's possible to be an expert at something but not be effective at teaching it, and I think failing to appreciate the difference often ends up leading to a lot of frustration that would otherwise be avoidable.
You are perfectly capable of parsing “3+2*4”, I don’t have to always write it as “MULTIPLY 2 BY 4, ADD 3”. Similarly you would get pretty mad if instead of “ld” or “lda” you’d have to type “load register” or “load accumulator” each time. Those concepts are trivial once you understood them, you don’t need a constant reminder of what they expand to.
Doing so might get slightly more people to “start using the thing” in the very first step, but might also let many people look for a more terse alternative. And we don’t have to cater to any person who is really unwilling to learn.
Overall I find it hard to buy that you actually had much trouble with $< and $@, at least in a way that you couldn’t have cleared up for yourself about 20 times in the time it took you to write out your comments.
I was hoping my introduction to Makefiles would be useful. If it was not, maybe I’m a bad educator after all.
$< is the name of the input (the file the rule depends on). If the rule depends on more than one input, it is the first one. So you write your dependencies as xxx.c xxx.h yyy.h zzz.h with the C file first. Mnemotechnically, it is "the thing that goes into the build".
$^ is the names of all the files the rule depends on. Think of it as a large hat above all the input file names.
Many things in make are clumsy and GNU make is a behemoth that does so much more than the original make that it is really hard to learn -- but these shortcuts have really good names!
(They are GNU make extensions, btw.)
For how often one typically writes/looks at build scripts, remembering the specifics about makefiles is quite likely to not be worth the effort.
Makefiles start becoming incredibly awful if you want to do anything outside of pure computations in the separate targets. Making the target directory requires either putting the mkdir call in each separate built thing (awfully redundant), introducing a whole another rule (and still adding it as a dependency on every rule that'll be outputting to said directory), doing some recursive make, or running it outside of rules (but putting it under a proper if statement is likely pain). That's a bunch of awful solutions to something that really really really should just be a single calls to mkdir in any sane system.
Another horrifically awful thing is if you want some reused data across rules (e.g. pkg-config results, current OS/arch). You can put them as globals, but that'll make them be calculated always, even if no build target needs them, unless you put them under an if, which is again non-trivial to get the condition correctly for. This becomes super bad if you do recursive make stuff.
More generally, makefiles start out simple, but doing anything that's not strictly simple is gonna be pain, at the very least requiring looking at docs of a thing you'll probably never use again, and debuggability is also very low (amusingly, '-p' and '--debug' break on recursive make, seemingly losing some variables; and '-n' requires copying out & running the recursive call to get to the actually important part; and afaict none of those help with figuring out why a thing didn't build).
And you're completely SOL if you wanted to do things like reorder compilations based on how long they previously took, or want a nice display of currently-compiling things.
I also have a lot of phony targets. It really enjoy just being able to type:
make restore kernel reset upload
Apparently .WAIT and .NOTPARALLEL exist, which I couldn't find last time I needed something around this. Though for this unfortunately `make -j4 foo .WAIT bar` isn't allowed, but at that point separate make invocations is fine.
make restore & (make -j 20 kernel && make reset upload)
---
Something skipped over by your introduction (and most guides to makefiles) which very quickly either makes makefiles less short or much more complicated is header files.
C files often depend on one or more header files that aren't system headers (this C compiler certainly does).
They could be added to each individual rule by hand:
codegen/codegen.o: codegen/codegen.c codegen/codegen.h linked_list.h libc.h
$(CC) -c -o $@ $<
HEADERS = codegen/codegen.h parser/parser.h parser/goto.h linked_list.h libc.h
codegen/codegen.o: codegen/codegen.c $(HEADERS)
$(CC) -c -o $@ $<
Or we could do something really fancy and get only the right header files added as dependencies for each compilation unit. That requires using a feature of the gcc/clang compilers (their preprocessors, really) that causes them to output a small makefile snippet that lists the exact header dependencies for a compilation unit (the '-MD' option). These snippets can then be included in the makefile. To make this look relatively clean requires some fancy macro use that is waaaay beyond what people can learn in ten minutes.
This is an extremely common use case and it is a bit of an embarrassment that GNU make doesn't handle it better.
It sure would be nice to have something like this:
codegen/codegen.o: !codegen/codegen.c
$(CC) -c $(CFLAGS) -o $@ $<
output: dependencies
<command(s) that use dependencies to create output>
Interesting, I thought that "make" was like multiplication tables, something you learn very early one as it's a super simple tool. Also I assumed it was part of any unix 101 class, together with bash and command line stuff. But probably it shows my age. Most people don't have any reason to learn these tools nowadays.
Whether this is true or not depends a lot on from which programming culture/background you come.
For example, a Makefile that does the same job as the build.py script in this project would be significantly smaller, simpler, and easier to read in several metrics that I'd reasonably call "objective" to a certain degree.
In fact, contrast the Makefile in that project: https://github.com/keyvank/30cc/blob/main/Makefile
With the build.py script: https://github.com/keyvank/30cc/blob/main/build.py
You need to know very little about Makefiles to make immediate sense of what that Makefile is doing, whereas you need to know much more about python to still not immediately see what the build.py script is doing. In fact, you will probably just "guess" that the python script is supposed to do a similar job only from its name before that.
And then the python script still does not do incremental builds at all!
Again, if it gets more complex that can change, but this is far away from that. It takes 10 or so minutes to learn enough about Makefiles to be productive with them, from scratch.
Left : right
Left : right
cp right Left
echo and so on. Left is now updated.
When it comes to the more "advanced" concepts (wildcards etc) there are slight differences between Make versions. And to bother with that is to get into the mindset which created the (argh) Automake and Autoconf systems (to begin with), so back in the day our company simply decided that we'll use GNU Make on every single system (we supported lots of various UNIX systems) and not bother with any of that (no SYSV Make, no BSD Make or anything), because GNU Make was and is available on anything and everything. Made life very simple back then.
The classic XKCD graph that plots life satisfaction against days-since-editing-xorg.conf could equally apply to days-since-thinking-about-autotools.
In fact one of the few times I've thought about autotools in the last decade was when a failing python-based build script inflicted similar frustration!
If I were forced to find one nice thing to say about autotools it would probably be that at least it doesn't assume an internet connection is always available, reliable and without cost.
Why isn't that also done by the Makefile? The only catch I could see is that you'd need to have it build to different output names, but that seems fine for what it is?
---
Also, I'm curious - did you find yourself having to constrain your use of C in order to make sure that the compiler could compile itself? Or does it implement everything you would use naturally anyways?
(And of course, congrats/bravo; super impressive:])