While I understand how finding a particular unique resource cached may allow to glean some privacy-infringing info, I don't see how detecting a common library downloaded from a site that provides it for everyone (like python.org) would help reveal anything.

If users might define a list of sites trusted for cross-site caching (like fonts.google.com and other known library CDNs), this could help cache quite some common resources without downsides to user's privacy.