Is it really a time sink? You have to have a build system anyway to link together your project itself, assuming it's bigger than a single-file hello world. One extra line in your build system should be the least time consuming part of adding a new foreign dependency--you still have to vet the code for security and figure out how to use it.

Unless you're using C++20 modules, you also have to deal with possibly including the header multiple times (slowing down builds), namespacing, macros potentially defined by the header, or a bunch of external/internal linkage edge cases. You only ever find out about these problems once it's too late to remove that library for a different one.