Not sure what you mean but typically you don’t need to track changes of libraries to that level. At least not in the context of a repo using those libraries. I am thinking of treating them like binary .dlls.

Source control is really good at compressing text files as they evolve over time, but isn’t optimized to handle binary assets. Since a single-line patch changes the entire zip archive, you’d risk growing the size of the repository based on the number of patches.