Some of them need high amounts of compression to be viable. Others don't have any concept of locality that one can map into a single dimension. For some, locality of changes is even an anti-feature.
"People" don't care what things are stored as. You could make GUI tools save to GraphViz's text based format (or heck PlantUML[0]) for graphs at least and it would work just fine.
For assets which have to be truly binary, Perforce offers versioning of those. I know it's not open source and costs money, but if it's really a need, it's worth it.
[0] - https://plantuml.com/
Edit: To clarify, if you don't need merging than you also don't need distributed VCS and centralized VCS will be sufficient. That is why I explicitly talked about distributed VCS.
However, I think the "fault" lies neither in the VCS nor file format. It simply shows that merging requires domain knowledge and that future VCS will need to cooperate with file formats to achieve that.