Re:Oxen.ai: Very Fast Unstructured Data Version Control
by yarg on Tuesday February 21, 2023 @07:21AM
(#34880276)
If you want blob support, you'd probably need to introduce format specific understanding of file structures.
You'd also need for the blob encoder to be exceptionally stable - if minor changes to the input lead to significant changes to output, you're going to have a lot of trouble.
Something like Google's Courgette (binary patch generation) could help to simplify things, but an old thread mentioned a patent suit.
Re:Oxen.ai: Very Fast Unstructured Data Version Control
by bfm on Tuesday February 21, 2023 @05:16PM
(#34888409)
Shameless plug. I open-sourced https://snapdir.org/ to explore how to solve these exact issues.
It's an early prototype to gather feedback around the ergonomics of the interface and the manifest format, and I would love to learn if people might find it useful enough for the use cases where LFS falls short.
A few starting points from personal experience would be
- Ensuring git-aware code editors don't try and mangle blob histories when they encounter them - Modern, high-performance blob downloads/version-resolution - Human-readable/debuggable meta-files, where it's easy to recover from bad states (instead of an opaque empty file with a GUID...?)