Even if I do have a catch-all, what tells me that it is a "library" exception? They might have thrown a generic std::exception with a vague string description. They might have thrown an "int". At some point what I really want is proper language and compiler support for those boundaries so I can know exactly where an exception came from, and can trap it at those boundaries (or ask the library maintainers to do so).
A function could just not use exceptions at all and just call abort() on error, what are you going to do then?
There isn't really a concept of library boundary in C++; you just have compilation units at best. At worst, you have a library that's implemented entirely as headers. What happens when the exception originates in your code but goes through the libraries call stack (because it was a call on a passed-in object or callback)? What you are asking for is likely impossible.
That's the correct way to do it; blanket ignoring all unhandled exceptions from a library is just very strange by comparison. If there is a specific exception that you want to ignore, by all means ignore it and document it heavily. But you don't know if every exception is irrelevant or potentially data corrupting.