The problem are timing attacks leaking the navigation history. An attacker can load a lib shared with some other site. Based on the load time he may then learn if the user visited the target site recently.

Can't we record the original download timings and replicate them artificially, with some noise added for privacy? Of course, with reasonable upper latency threshold (derived from overall network performance metrics) to prevent DoS on clients.

While this won't improve load times, it can save bandwidth, improving user experience on slow connections. And most popular hand-selected library versions can get preloaded by the browser itself and listed as exceptions with zero added latency.

Also, this way the networking telemetry would gain a meaningful purpose for end users, not just developers.

My understanding had been that you get a double whammy of there being too many versions of the “common” dependencies, so it is already in-effect a per-site cache (or nearly so), but that uniqueness also means a fingerprint can be established with fairly few latency tests

How would you know the original site if it's a common asset?

No need. Just make a custom asset referenced only by those sites, and get the browser to cache it.

There is some level of data exposure even if you can't tell which site, specifically, a user has visited. You can tell that "this user has been to at least one page that used X".

Depending on what the common resource is, most people wouldn't necessarily care but there may be some cases where they would.

If I can tell that you used facebook login at one point, even though it is used by a good chunk of the web, I could tailor a phish targeted at FB credentials for example.

Apart from tracking, the other concern is reliability — the server may lose or change the file, and this may not be noticed when clients use the hash, until the hash stops being popular, and you're left with a broken site.

If you change the file the hash changes

The hash is sent separately from the file. For it to change in the HTML and other places linking to it, there has to be some mechanism to update the links, and that is the part that can get out of sync or bitrot.

Which is part of the security of the hash. When I embed a script on my site I get the hash at the time I test it. If the content changes without my knowledge it will not allow a replacement from the source. Assets are not overly cacheable if they don't change the location when the file changes.