Does it really blow up compile time? Translation units and optimization are usually where compile time is spent. Having fewer but fatter translation units helps compile times tremendously while most of the time per translation unit is spent in stages beyond the source compiling in LLVM.

Boost is really the only example I can think of where small utility comes at the expense of huge compile time increases.

Header-only libraries aren't generally single-header-only. That's a bit of an extreme.