I think there's just a bunch of history here--print is the use case that you really want PDF to get correct, and so you take existing print pipelines which discard semantics and add PDF as a new backend. This is the easy/lazy way to get PDF working, and there are also tons of PDF pipelines are just stapled onto the end of a print pipeline as if PDF were another type of printer.
You'll see a night and day difference between one of those pipelines and something more suited to processing a document with semantics, like FrameMaker. FrameMaker seems to be a bit more niche now that print documentation is less mainstream.
I can understand why the PDF ecosystem is a bit frustrating, but I still prefer it when I'm reading academic papers explaining some algorithm or mathematical concept.
When the alternative is that they don't offer a non-proprietary export format at all, I'll take it. Even a poor PDF output is better for data portability than having your data forever trapped in a proprietary platform that can disappear or get discontinued at any time without a migration path.
OCR applications are able to make PDFs from scanned images accessible and searchable (and with the right spaces) precisely because they embed their output into that layer of the PDF meant for the text itself, not its presentation.
If you are generating a PDF with text you can render a string with spaces and let the glyph layout sort it out, or render individual characters, or draw those characters by hand and use the invisible font trick (because your font is not licensed for embedding). It’s not that you’re using different conceptual layers of the PDF, you’re just choosing how to render your content.
Accepting a potentially inferior machine-readable plaintext when copy and pasting is the price we pay for having something like that available.
The same can be said about P.D.F. files.
I recently found out why so often when copying from them, spaces are either inserted doubly, or removed: the reason is that the format has no understanding of spaces, only of distance between characters and is completely about præsentation, lacking semantics in any way.
When copying, the reader simply applies a heuristic of what distance between characters constitutes a space, and is often wrong.