> Maybe if you use a CDN, so hopefully the user might have a local cache of it from somewhere else, that can be avoided?

Browsers partition caches by origin now for privacy reasons, so this is no longer possible. If the user doesn’t have it cached from your website, they don’t have it cached.