News for hackers. Stuff that matters.
Re:Eden by m12k on Tuesday April 12, 2022 @03:41PM (#31007351)
I'm also happy with git, but there's 3 main things that could improve on git IMO:

1) Better handling of large files than git-lfs. As in 10+ GB repos. This is needed for game development (currently they tend to use Perforce or PlasticSCM)

2) Sparse checkout via file system integration (like Eden has)

3) Build system integration, so unchanged files and modules don't even need to be fetched to be compiled, because cached builds can be fetched from a build server instead (requires proper modularization, so e.g. C++ macro expansion doesn't just prevent anything from being cacheable)

These are all features that primarily have value for repos that push the limits on size, like big monorepos (with a huge amount of files) or game development (with big asset files). But get it right, and you could massively cut down the time it takes to check out a branch and build it.

Re:Eden by kvnhn on Tuesday April 12, 2022 @06:26PM (#31008992)
This is by no means a perfect match for your requirements, but I'll share a CLI tool I built, called Dud[0]. At the least it may spur some ideas.

Dud is meant to be a companion to SCM (e.g. Git) for large files. I was turned off of Git LFS after a couple failed attempts at using it for data science work. DVC[1] is an improvement in many ways, but it has some rough edges and serious performance issues[2].

With Dud I focused on speed and simplicity. To your three points above:

1) Dud can comfortably track datasets in the 100s of GBs. In practice, the bottleneck is your disk I/O speed.

2) Dud checks out binaries as links by default, so it's super fast to switch between commits.

3) Dud includes a means to build data pipelines -- think Makefiles with less footguns. Dud can detect when outputs are up to date and skip executing a pipeline stage.

I hope this helps, and I'd be happy to chat about it.

[0]: https://github.com/kevin-hanselman/dud

[1]: https://dvc.org

[2]: https://github.com/kevin-hanselman/dud#concrete-differences-...

Re:Eden by avar on Wednesday April 13, 2022 @04:49AM (#31012469)
I'd be curious to see if you've tried git-annex, I use it instead of git-lfs when I need to manage big binary blobs. It does the same trick with a "check out" being a mere symlink.
Re:Eden by kvnhn on Wednesday April 13, 2022 @05:05PM (#31020255)
I haven't used it, no. Around the time Git LFS was released, my read from the community was that Git LFS was favored to supersede git-annex, so I focused my time investigating Git LFS. Given that git-annex is still alive and well, I may have discounted it too quickly :) Maybe I'll revisit it in the future. Thanks for sharing!
Re:Eden by avar on Thursday April 14, 2022 @08:36AM (#31026086)
Neither is favored, git-annex solves problems that git LFS doesn't even try to address (distributed big files), at the cost of extra complexity.

Git LFS is intended more for a centralized "big repo" workflow, git annex's canonical usage is as a personal distributed backup system, but both can stretch into other domains.

In this case git-annex seems to have a feature that git LFS doesn't have that would be useful to you.

Re:Eden by maccard on Wednesday April 13, 2022 @05:43AM (#31012744)
I work in games, and we use PlasticSCM with Unreal Engine.

> Better handling of large files than git-lfs.

PlasticSCM does really really well with binary files.

> Sparse checkout via file system integration

Windows only [0] but Plastic does this. I've been working through some issues but it's usable as a daily driver with a sane build system.

> Build system integration

UnrealBuildTool at the same time is both the coolest and most frustrating build systems I've ever used. It's _not_ a general purpose build system for anyone and everyone, it's tailored to Unreal, it's slow to run the actual build system, the implementation is shaky at times, but some of it's features are incredible. Two standout features are Unity Builds, and Adaptive Unity builds. Unity builds are common now [1], but adaptive unity is a game changer. It uses the Source control integrations to check what files are modified and removes them from the unity blobs, meaning that you're only ever rebuilding + relinking what's changed, and get the best of both worlds.

[0] https://blog.plasticscm.com/2021/07/dynamic-workspaces-alpha... [1] https://en.wikipedia.org/wiki/Unity_build

Re:Eden by thomascgalvin on Tuesday April 12, 2022 @05:11PM (#31008284)
ClearCase (claimed to) support points two and three way back in the 90s. It never really worked right; it was always trying to "wink-in" someone else's binary, despite the fact that it was built using a different version of the source.
Re:Eden by webmaven on Tuesday April 12, 2022 @05:03PM (#31008204)
> These are all features that primarily have value for repos that push the limits on size, like big monorepos (with a huge amount of files) or game development (with big asset files). But get it right, and you could massively cut down the time it takes to check out a branch and build it.

Would this be a good fit for large machine learning data sets?

Every time a new model comes out that touts having been trained on something like a quarter of a billion images, I ask myself "how the heck does someone manage and version a pile like that?"

Re:Eden by zem on Tuesday April 12, 2022 @08:14PM (#31009743)
would you ever need to version individual images? at a high level, you could version the collection as a whole by filtering by timestamp, and moving deleted files to a "deleted" directory, or maintaining filename and timestamps in a database. I'm sure there are lots of corner cases that would come up when you actually tried to build such a system, but I don't think the overall scheme needs to be as conceptually complex as source code version control
Re:Eden by webmaven on Wednesday April 13, 2022 @12:26PM (#31016659)
> would you ever need to version individual images?

Potentially. It would depend on the image format and what metadata is (or can be) included in the file.

Re:Eden by bastardoperator on Tuesday April 12, 2022 @06:28PM (#31009005)
Not handling large binary files is a feature from my perspective. Git is for source code and treating it like a disk or a catch all for your project is how people get into scaling trouble. I don't see any reason why versioned artifacts can't be attached to a build, or better, in a warm cache. I get that it's easier to keep everything together, but we can look at the roots of git and it becomes fairly obvious, Linus wasn't checking in 48GB 4K mpeg files for his next FPS.
Re:Eden by maccard on Wednesday April 13, 2022 @05:46AM (#31012774)
> I don't see any reason why versioned artifacts can't be attached to a build, or better, in a warm cache

Because now you've got two versioning systems with different quirks and tools that behave slightly different that _must_ be kept in sync.

> I get that it's easier to keep everything together,

Easier is a bit of an understatement. Putting binary files outside of the project in another versioning system means you need two version control systems, _plus_ custom tooling to glue the two together. Also, the people who interact with these assets primarily are not technical at all, they're artists or designers, and they're the most likely to have issues with falling between the cracks.

Re:Eden by 411111111111111 on Tuesday April 12, 2022 @05:14PM (#31008303)
> Build system integration

It would be a game changer if any SCM actually manages to achieve this without needing multiple developer teams just for maintenance.

That would probably break gits current dominance within a relatively short timeframe... But I don't think it'll happen within the foreseeable future,

Re:Eden by avar on Tuesday April 12, 2022 @05:59PM (#31008759)
If you want an SCM with direct build system integration there's always GNU make with its support for RCS :)

More seriously, can you describe what "build system integration" would look like? Basically like what GNU make does with RCS? I.e. considering the SCM "files" as sources.

How would such a system build a dumb tarball snapshot?

Re:Eden by xmcqdpt2 on Tuesday April 12, 2022 @08:34PM (#31009881)
At my work we have such a system for our monorepo. Basically for each compilation unit (basically subdirectories), it takes the compilation inputs and flags etc and checks them against a remote cache. If it matches, it just pulls the results (binaries but also compiler warnings) from the remote instead of compiling anything. A new clean build is made every few commits to the main branch to populate the cache.

In practice it means that if I clone the repo and build it doesn't ever invoke the compiler, so that it takes only a few minutes instead of an hour.

Bazel has a similar system but I haven't used it myself,

https://docs.bazel.build/versions/main/remote-caching.html

Re:Eden by avar on Wednesday April 13, 2022 @04:44AM (#31012441)
That seems really useful, but how is this different from what ccache and various other compilation caches that cache things as a function of their input do?

The GP talks about "build system integration" being "a game changer", and I can see that being a useful thing for e.g. "priming" your cache, but that's surely just a matter of say:

    ccache-like-tool --prime $(git rev-parse HEAD:)
I.e. give it an arbitrary "root" key and it would pre-download the cached assets for that given key, in this case the key is git-specific, but it could also be a:

    find -type f -name '*.[ch]' -exec cat {} \; | sha1sum | cut -d' ' -f1
Or whatever.
Re:Eden by maccard on Wednesday April 13, 2022 @05:48AM (#31012783)
> how is this different from what ccache and various other compilation caches that cache things as a function of their input do?

Vertical integration, basically. There is value in these things working, and working well. As an example of the issues with ccache etc, I've yet to find a compilation cache that works well on windows for large projects.

Re:Eden by xmcqdpt2 on Wednesday April 13, 2022 @07:40PM (#31021686)
I've never worked on a project with a large remote ccache but I would guess it would be pretty much the same yes.

The "automation" of our in-house system is what really makes the difference but then again we have a team of developers that focus on tooling so it's not so much automated as it is maintained...

Re:Eden by hackerbrother on Tuesday April 12, 2022 @09:29PM (#31010242)
Is (3) doable with the abstractions over Git that Git{la,hu}b provide?