News for hackers. Stuff that matters.
Re:A Pipeline Made of Airbags by saagarjha on Friday September 25, 2020 @04:10AM (#24587696)
I work with a project that takes an hour to build because it’s thousands of C++ files. What would you suggest I do to improve my build times?
Re:A Pipeline Made of Airbags by pjc50 on Friday September 25, 2020 @05:47AM (#24588218)
Surprisingly, "concatenative builds" where you chuck all the files in a single file and compile that, tend to be a lot faster.

distcc/ccache also works. Years ago we had a rack full of machines devoted to this. Requires a little tuning but lets you do -j 50. Of course, that was in the days of dual-core machines and these days you can buy a lot of cores. Bottleneck is probably writing intermediate files to disk and flushing them.

Untangling the inter-dependencies that cause all the rebuilding is very hard and often forces design changes you don't want, such as use of the "pimpl" idiom.

Re:A Pipeline Made of Airbags by rob74 on Friday September 25, 2020 @06:40AM (#24588468)
Even without having programmed in C/C++ for years, it doesn't sound very surprising that "concatenative builds" are faster - I assume it has something to do with the preprocessor having exponentially less work the fewer files you have?
Re:A Pipeline Made of Airbags by rlaanemets on Friday September 25, 2020 @07:50AM (#24588808)
Our QT project had about 400 compilation units (header+source, compiled to a separate object file). Many of them were 100-200 lines of code only. Full build used to take 40 minutes. Each compilation unit pulls in about 40k lines of code from Qt through headers. So the compiler has to process at least 400*40k = 16M lines of code. Concatenative build only processes app code (about 50k) plus about 60k of cumulated framework code. This gives about 100 times speedup. Our concatenative build took indeed less than 1 minute.
Re:A Pipeline Made of Airbags by saagarjha on Saturday September 26, 2020 @10:41PM (#24603785)
This is with unified builds :(
Re:A Pipeline Made of Airbags by coding_unit_1 on Friday September 25, 2020 @04:14AM (#24587721)
Do you need to recompile all multi-thousand files every time you make a change? Modularising the build would be my first step.
Re:A Pipeline Made of Airbags by rkangel on Friday September 25, 2020 @05:39AM (#24588177)
A good build system and caching. By "a good build system" I mean a hermetic one like Bazel that can be confident of knowing the inputs to each build step so that they can be run or not as appropriate. Sometimes you'll touch a core header file and a lot of things will rebuild, but it should still be a lot better.
Re:A Pipeline Made of Airbags by magicalhippo on Friday September 25, 2020 @04:48AM (#24587899)
A buddy slashed his CI build times by using a RAM disk. But yeah, usually faster to avoid compiling things.
Re:A Pipeline Made of Airbags by froj on Friday September 25, 2020 @05:19AM (#24588049)
We have a similar situation and use ccache and distcc is setup on each dev's workstation. It also helps to use forward declarations instead of including header files whenever possible.