You can see the brief list of improvements here:
https://martine.github.io/ninja/manual.html#_comparison_to_m...
(Within Ninja this manifests concretely as needing to represent the build graph as a bipartite graph between files and commands. Though I don't know make's internal representation, I imagine the straightforward implementation is as a graph between files and I can imagine why it'd be difficult to make multiple outputs work in that case.)
However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all.
Because we cared more about performance than other things we maybe were a little crazier about shaving off milliseconds than other systems, so it might be hard to make make fast enough. But (as the Ninja manual suggests) this may only matter for very large (Chrome-sized) projects. You can read a chapter about some of the performance work done on Ninja here: http://www.aosabook.org/en/posa/ninja.html
Not exactly. I'm asking if an implementation of the portable subset of Make people actually use (pointedly excluding GNU Make extensions) could be made nearly as fast as Ninja. That would have the advantage of running existing makefiles, as long as those makefiles weren't written specifically for GNU Make.
When you write "could be made nearly as fast as Ninja", I thought you meant you would change the code of Make or write a new tool. If you don't write a new tool and just use a subset of Make's functionality for your own Makefiles I'd expect those to be faster than your standard Makefiles, sure.
But "existing makefiles" typically use a mixture of GNU make-isms and other things that are not GNU-specific but still slow (e.g. from a skim of the BSD make manual I see lazy variable expansions and control structures). So if you're talking about implementing a subset of Make you're talking about something that's unlikely to be compatible with Makefiles seen in the wild. And then if you're not compatible with Makefiles seen in the wild you've effectively written an incompatible faster subset of Make, which brings us back to what Ninja is. (In case it's not clear, in Chrome's case the same code was used generate Makefiles and Ninja files with some relatively minor output differences -- the tools are that close.)
Perhaps there's some "common" subset of Make that covers some high percentage of build files seen in the wild that could be made faster. That could be valuable to organizations who want faster builds without changing anything. I vaguely recall seeing some commercial software that did this even -- as I recall, their value add was that they had tooling that would figure out how to run your Makefiles in parallel; though Make supports parallel execution, apparently this business found enough customers by just targeting those with underspecified Makefiles that weren't parallel safe and then fixing them!
Anyway, all of this is kinda moot because for 99% of projects Make is plenty fast. I even use Make myself for all my personal projects! In Chrome's case (what we wrote Ninja for) there's so much code that the build files themselves are over ten megabytes of text, so to parse that quickly you're at the level of worrying about avoiding memory allocations in the lexer, which is beyond what most people would care about.
I was partly asking because a fast make-compatible build tool seems easier to switch to, and partly because I wondered how much of Ninja's performance depends on dropping the slow features of make versus adding new capabilities or new syntax.
Okay, now I'm wondering if I could get Make to generate a Ninja file.
%.tab.h %.tab.c: %.y
stuff
Using pattern matching forces make (GNU make) to correctly treat this as one rule which outputs multiple files, instead of the default of it being multiple rules which each output one file. It's not perfect, but it is a simple fix for some of those issues.If you run without parallelism (the default, -j1), it'll work fine, since make will run the rule for one target, and then when it comes to the second one, make will notice that it's already up-to-date (based on file timestamp).
If you run with -j4 or something, it make will likely run the rule twice, simultaneously. That may or may not be a bad problem depending on how the rule writes the output file.
EDIT: man I must have really rushed reading your post... as you say, this does appear to be a heuristic for GNU Make when it's a pattern rule. clearly make is tricky and I end up learning stuff every time I use make...
Anyway, take a look at https://github.com/spotify/luigi, it is basically make-like tool geared toward data pipelines. We are experimenting with it for our data pipeline (which is quite sizeable, ~30TB/day gzipped, thousands of files, a bunch of processing steps) and while it lacks some things, especially in the UI, this approach seems to work quite well.
Allowing steps to partially succeed seems like adding quite a lot of complexity, and I'm not sure where it would be beneficial.
With Electric Make you can mark the rule as producing multiple outputs by adding "#pragma multi" above the declaration. See http://blog.melski.net/2013/01/01/pragma-multi-and-rules-wit....
Disclaimer: I'm the Chief Architect of Electric Make
Jeez, you know, this is the sort of thing that I would hate to get somewhat deep into using make, only to then discover.
I often see hackers lamenting that a plethora of build tools exist instead of everyone just using make, but I'm not so sure that this state of affairs is less preferable. Maybe it's better if the build tool understands some things about the platform it's building for.
%.tab.c %.tab.h: %.y
bison -d $<
[0] http://www.gnu.org/savannah-checkouts/gnu/make/manual/html_n...I did have one percieved speed problem, but this scons calculating file checksums to see if it needed to re-build targets; this is usually a good thing, but if you're working with large files it might be better to use the MD5-timestamp Decider instead.
If a multiple-output step fails, I want the outputs to be deleted, and the rest of the dependencies to keep working so I don't have to rebuild everything from scratch. Whatever you're suggesting with dummy targets sounds like it's certainly not going to track the dependencies correctly.
I already know that directories aren't tracked correctly as dependencies.
As for workarounds... this is hacker news, is it not?
One deficiency I've come across, which seems to be well-known, is the M:N problem -- where one step takes M input files and has N output files. Make rules seem to expect to have only 1 output, and the workarounds like .SECONDARY prevent some features of Make from working correctly.
I've also seen this limitation in many of the fancy new build systems that get posted here on HN.
Is there a build system, a modification to Make, or anything out there that does keep track of builds with multiple outputs? Not an I/O-guzzling MapReduce framework, please.