News for hackers. Stuff that matters.
Re:Tup – an instrumenting file-based build system by kazinator on Tuesday August 16, 2022 @12:48PM (#32485422)
There shouldn't be any .d files in a from-scratch clean build. They get generated during compilation and are not needed initially because every object file is out of date due to its nonexistence.

However, they do add a lot of bulk once made and increase with the size of the project.

A lot of the information is redundant. Among the files you may find many pairs which only differ in the principal file, the headers being the same:

   foo.o: foo.c $SOME_HEADERS

   bar.o: bar.c $SOME_HEADERS
In theory, if we have GNU Make, we should be able to compress these into static pattern rules. Concretely, in the above case, the pair condenses to the static pattern rule:

   foo.o bar.o: %.o: %.c $SOME_HEADERS
When, we check whether, say, bar.o is out of date, we find that it as a target in the left side of the rule. Then in the middle, the name matches the %.o target pattern, identifying the stem as "bar", and from there the full dependency rule is instantiated with all the prerequisites.

To generalize that, we partition the $(OBJECTS) into subsets based on which patterns they belong to. Hopefully there are a lot fewer subsets than files.

Then for each subset we do

  # The a.h b.h subset
  $(OBJS_0): %.o: %.c a.h b.h

  # The a.h d.h subset
  $(OBJS_1): %.o: %.c a.h d.h
  
and so on. The only problem is managing these subsets (in particular, incrementally) in such a way that more time isn't spent doing that than time saved loading the rules.

Stale dependencies can be troublesome when header files are renamed or deleted; this sort of scheme could make it worse.

One way to handle these subsets incrementally would be to keep them in some separate database outside of the Makefile. Whenever a file is compiled and the raw .d is generated, we run a script which looks up the header signature and matches it to a subset. Then it assigns that file to the correct subset (perhaps deleting it from an existing one). A step at the start of the build would regenerate the dependency file seen by make if the database indicates that it's out of date.

Re:Tup – an instrumenting file-based build system by drothlis on Tuesday August 16, 2022 @02:56PM (#32487166)
Ninja stores the header dependencies in a database and then deletes the “.d” file. As far as I know it doesn’t do your deduplication – presumably because it hasn’t been necessary; reading and parsing hundreds of “.d” files was the slow part.