Re:Tup – an instrumenting file-based build system
by drothlis on Tuesday August 16, 2022 @02:56PM
(#32487166)
Ninja stores the header dependencies in a database and then deletes the “.d” file. As far as I know it doesn’t do your deduplication – presumably because it hasn’t been necessary; reading and parsing hundreds of “.d” files was the slow part.
However, they do add a lot of bulk once made and increase with the size of the project.
A lot of the information is redundant. Among the files you may find many pairs which only differ in the principal file, the headers being the same:
In theory, if we have GNU Make, we should be able to compress these into static pattern rules. Concretely, in the above case, the pair condenses to the static pattern rule: When, we check whether, say, bar.o is out of date, we find that it as a target in the left side of the rule. Then in the middle, the name matches the %.o target pattern, identifying the stem as "bar", and from there the full dependency rule is instantiated with all the prerequisites.To generalize that, we partition the $(OBJECTS) into subsets based on which patterns they belong to. Hopefully there are a lot fewer subsets than files.
Then for each subset we do
and so on. The only problem is managing these subsets (in particular, incrementally) in such a way that more time isn't spent doing that than time saved loading the rules.Stale dependencies can be troublesome when header files are renamed or deleted; this sort of scheme could make it worse.
One way to handle these subsets incrementally would be to keep them in some separate database outside of the Makefile. Whenever a file is compiled and the raw .d is generated, we run a script which looks up the header signature and matches it to a subset. Then it assigns that file to the correct subset (perhaps deleting it from an existing one). A step at the start of the build would regenerate the dependency file seen by make if the database indicates that it's out of date.