Optimising compression and caching zoom levels does make a big difference. But it is exactly what is missing in a significant number of PDF files in the wild. This is particularly common in engineering drawings/maps and they can be slow to open on any machine. The point of my original comment was that client side code sitting is isolation will always have certain limitations regardless of how it is implemented.
Usually when the PDF is silly big, someone just didn't do a good job preparing it.