Re:A Pipeline Made of Airbags
by saagarjha on Friday September 25, 2020 @04:10AM
(#24587696)
I work with a project that takes an hour to build because it’s thousands of C++ files. What would you suggest I do to improve my build times?
[ Parent
| on HN ]
Surprisingly, "concatenative builds" where you chuck all the files in a single file and compile that, tend to be a lot faster.
distcc/ccache also works. Years ago we had a rack full of machines devoted to this. Requires a little tuning but lets you do -j 50. Of course, that was in the days of dual-core machines and these days you can buy a lot of cores. Bottleneck is probably writing intermediate files to disk and flushing them.
Untangling the inter-dependencies that cause all the rebuilding is very hard and often forces design changes you don't want, such as use of the "pimpl" idiom.
Even without having programmed in C/C++ for years, it doesn't sound very surprising that "concatenative builds" are faster - I assume it has something to do with the preprocessor having exponentially less work the fewer files you have?
Re:A Pipeline Made of Airbags
by rlaanemets on Friday September 25, 2020 @07:50AM
(#24588808)
Our QT project had about 400 compilation units (header+source, compiled to a separate object file). Many of them were 100-200 lines of code only. Full build used to take 40 minutes. Each compilation unit pulls in about 40k lines of code from Qt through headers. So the compiler has to process at least 400*40k = 16M lines of code. Concatenative build only processes app code (about 50k) plus about 60k of cumulated framework code. This gives about 100 times speedup. Our concatenative build took indeed less than 1 minute.
This is with unified builds :(
Re:A Pipeline Made of Airbags
by coding_unit_1 on Friday September 25, 2020 @04:14AM
(#24587721)
Do you need to recompile all multi-thousand files every time you make a change? Modularising the build would be my first step.
A good build system and caching. By "a good build system" I mean a hermetic one like Bazel that can be confident of knowing the inputs to each build step so that they can be run or not as appropriate. Sometimes you'll touch a core header file and a lot of things will rebuild, but it should still be a lot better.
Re:A Pipeline Made of Airbags
by magicalhippo on Friday September 25, 2020 @04:48AM
(#24587899)
A buddy slashed his CI build times by using a RAM disk. But yeah, usually faster to avoid compiling things.