A single file? Highly unlikely.

But three specific files can already be pretty unique. I chart.js with two specific plugins in my toy project, and I'm willing to bet that no one else on the world uses the exact same set and version configuration.

Exactly, but a third party can't see that set from the cache, they see the union of every website recently visited. They would see hundreds of files from many websites and if only one of those uses one of the three files yours does, it's impossible to tell for sure without a file that isn't used anywhere else on the Web. Your site uses A+B+C, site 2 uses A+D+E, site 3 uses B+F, site 4 uses C. The cache contents is A+B+C+D+E+F+... did the user visit your site? It's like trying to get individual pictures out of a single piece of film that was exposed multiple times - you can make some guesses and rule some possibilities out, but nothing other than that will be conclusive.