Unless you're using the unversioned ("latest") jQuery version on your CDN, the browser cache hit rate will be way less than 95%. I've seen some estimates at less than 20% (can't find the link at the moment).
And for various reasons, it's not a terribly good reason to use the "latest" jQuery version with no version, unless you're prepared to deal with sudden code breakage.
> then there's a 95% your users already have it cached.
In reality, there's a less than 5% chance that your users will have the right version cached, but a 95% chance that your users will notice when the public CDN is over capacity.
I mean, it's like less than 30kB minified+gzipped, and if you load it from a CDN, then there's a 95% your users already have it cached.
I guess it's technically a "bloat" but unless all your images are gzipped SVGs, it's almost certainly one of the smaller elements in your page.