Header only works for some things (particularly things that require no globals, singletons, etc) and that's a valid concern. Saying header-only = long term technical debt always (or even most of the time) feels like an assertion because I've only heard hypotheticals around why it is bad.
Unless you're using C++20 modules, you also have to deal with possibly including the header multiple times (slowing down builds), namespacing, macros potentially defined by the header, or a bunch of external/internal linkage edge cases. You only ever find out about these problems once it's too late to remove that library for a different one.
Could, should. IRL Docker became a thing mostly because of the hassle it is to do so in C.
Also, now have to prefix the hell out of it, as even anonymous namespace won't work. And not only that.
Generating rpm or deb files is hardly the end of the world, there are even package generators available.
Long term technical debt