I think there are two issues. In C++, a header only library makes sense due to templates, which don't produce object code before specialization. In C, I have seen the same with entirely macro-based "libraries" that are really simulating templates in order to make abstract data structures.
The second case though, my theory is that there is a new generation of programmers who don't understand the traditional compile and link phases because they are used to other languages. It doesn't fit their expectation of how it should work so they bend the tool to their expectation instead of figuring out the old way. The very fact that people on here are saying a programming language should have a "standard package manager" demonstrates the cultural divide: this sounds a little nutty to an old time C person.
The disadvantage (and the reason that I'm personally not a fan) is that they bloat compile time. If it's a library you compile it once, and then it gets linked repeatedly, but a header only library is compiled every time any CPP file that includes it is compiled.
It may be that you have used a lot of template-based headers, which may compile nearly every time because they are literally creating new code every time a new combination of template parameters is given.
You are correct that I'm mostly talking about template heavy header files. There is a strong correlation between template based header files and header only libraries. The matter) latter generally means the former.
but are you going to mmap stuff in all your .cpps ? Most of the time when I use an external library, it does not get out of a single implementation file, so it being header only does not really make it worse
Once the infrastructure is sorted out, header-only has mainly disadvantages, like making it harder to separate interface from implementation.
There's a good talk on the topic: https://youtu.be/sBP17HQAQjk