News for hackers. Stuff that matters.
Re:Fossil versus Git by dognotdog on Saturday September 23, 2023 @07:32AM (#37622304)
I keep coming back to fossil again and again, despite git having a huge pull because of the easy publishing and collab on github/gitlab.

Just the other day I was starting an exploratory project, and thought: I'll just use git so I can throw this on github later. Well, silly me, it happened to contain some large binary files, and github rejected it, wanting me to use git-lfs for the big files. After half an hour of not getting it to work, I just thought screw it, I'll drop everything into fossil, and that was it. I have my issue tracker and wiki and everything, though admittedly I'll have some friction later on if I want to share this project. Not having to deal with random git-lfs errors later on when trying to merge commits with these large files is a plus, and if I ever want to, I can fast-export the repo and ingest it into git.

Re:Fossil versus Git by prepend on Saturday September 23, 2023 @08:12AM (#37622583)
It is extremely rare that I have a file over 100MB.

I also think it’s one of those situations where if I have a giant binary file in source control “I’m doing it wrong” so git helps me design better.

It’s like in the olden days when you couldn’t put blobs directly in a row so databases made you do your file management yourself instead of just plopping in files.

I like git. I don’t like giant binary files in my commit history. It’s cool that you like fossil, but I don’t see this as a reason for me to use it.

Re:Fossil versus Git by dwpdwpdwpdwpdwp on Saturday September 23, 2023 @08:43AM (#37622751)
Source control is all about managing diffs. Large files are fine, binary doesn’t make sense. Most of the time binary file diffs aren’t human readable.

I store binary files outside of git but keep build logs containing binary file CRCs on git

Re:Fossil versus Git by tom_ on Saturday September 23, 2023 @09:12AM (#37622945)
Most binary files that people want to store in a VCS are stuff like .psd, .xlsx, .docx, and the like - data that's created by people by hand, but not stored as text.
Re:Fossil versus Git by prepend on Saturday September 23, 2023 @10:30AM (#37623567)
Xlsx and docx are just zipped up xml text. You can store it as text if you like and I think there are many git modules to handle this. But the xml isn’t really that diffable so I don’t bother.
Re:Fossil versus Git by masklinn on Saturday September 23, 2023 @09:45AM (#37623162)
> Source control is all about managing diffs. Large files are fine, binary doesn’t make sense

In git, diffs are literally just a UI thing.

Re:Fossil versus Git by wavemode on Saturday September 23, 2023 @10:11AM (#37623375)
That's not really true, is it? Surely Git does have an internal concept of diffing changes, specifically so it knows whether two commits can be merged automatically or if they conflict (because they changed the same lines in the same file).
Re:Fossil versus Git by masklinn on Saturday September 23, 2023 @10:58AM (#37623833)
> That's not really true, is it?

It is.

> Surely Git does have an internal concept of diffing changes

Not in the data model. Packing has deltas, but they're not textual diffs, and they would work fine with binary data... to the extent that the binary data doesn't change too much and the delta-ification algorithms are tuned for that (both of which are doubtful).

> specifically so it knows whether two commits can be merged automatically or if they conflict (because they changed the same lines in the same file).

Conflict generation & resolution is performed on the fly.

Re:Fossil versus Git by IshKebab on Saturday September 23, 2023 @08:54AM (#37622838)
You didn't put blobs directly in the database because of annoying database limitations, not because there's a fundamental reason not to.

It's the same with Git. Don't put large files directly in Git because Git doesn't support that very well, not because it's fundamentally the wrong thing to do.

There should be a name for this common type of confusion: Don't mistake universal workarounds for desirable behaviour.

Re:Fossil versus Git by prepend on Saturday September 23, 2023 @10:33AM (#37623590)
The fundamental reason had to do with how rdbms structured its pages of data and having arbitrary sized blobs directly in the record broke the storage optimization and made performance tank.

It was a design constraint back in the day.

I haven’t looked at this in decades, but I think now it’s all just pointers to the file system and not actually bytes in the record.

So it was fundamentally the wrong thing to do based on how databases stored data for performant recall.

But that’s back when disks were expensive and distributed nodes were kind of hard.

Re:Fossil versus Git by sgarland on Saturday September 23, 2023 @12:20PM (#37624630)
> I think now it’s all just pointers to the file system

It depends. InnoDB, assuming the DYNAMIC row type, will store TEXT/BLOB on-page up until 40 bytes, at which point it gets sent off-page with a 20 byte pointer on-page. However, it comes with a potentially severe trade-off before MySQL 8.0.13: any queries with those columns that would generate a temporary table (CTEs, GROUP BY with a different ORDER BY predicate, most UNIONS, many more) can’t use in-memory temp tables and instead go to disk. Even after 8.0.13, if the size of the temp table exceeds a setting (default of 16 MiB), it spills to disk.

tl;dr - be very careful with MySQL if storing TEXT or BLOB, and don’t involve those columns in queries unless necessary.

Postgres, in comparison, uses BYTEA as a normal column that gets TOASTed (sent off-page in chunks) after a certain point (I think 2 KiB?), so while you might need to tune the column storage strategy for compression - depending on what you’re storing - it might be fine. There are some various size limits (1 GiB?) and row count limits for TOAST, though. The other option is with the Large Binary Object extension which requires its own syntax for storage and retrieval, but avoids most of the limitations mentioned.

Or, you know, chuck binary objects into object storage and store a pointer or URI in the DB.

Re:Fossil versus Git by fmajid on Saturday September 23, 2023 @09:53AM (#37623224)
In the age of Large Language Models, large blobs will become the rule, not the exception. You’re not going to retrain models costing $100M to build from scratch because of the limitations of your SCM.
Re:Fossil versus Git by prepend on Saturday September 23, 2023 @10:28AM (#37623546)
I don’t store those in my scm. It’s not a limitation of my scm that I can’t store a 20gig model directly in the repo.

So you’re right, I’m not going to retrain models costing $100M because of SCM limitations. That’s because I don’t have any SCM limitations.

Re:Fossil versus Git by otp209 on Saturday September 23, 2023 @12:38PM (#37624844)
> if I have a giant binary file in source control “I’m doing it wrong” so git helps me design better

Your VCS should not be opinionated, that is not its job

Re:Fossil versus Git by mariusor on Saturday September 23, 2023 @02:08PM (#37625771)
I fail to understand people that can't be bothered to empathize with other use cases than their own. Game development usually has a large number of binary assets that need to be in source control, does that sound like a reasonable use, or are they also doing it wrong?
Re:Fossil versus Git by Gibbon1 on Saturday September 23, 2023 @03:09PM (#37626369)
GF is working for a startup doing a game. They were using git and dumped it because it just cannot deal. Also the content people found it annoying without providing any value what so ever.
Re:Fossil versus Git by mgaunard on Saturday September 23, 2023 @08:17AM (#37622614)
if you have large files in your repository, you have a design problem.
Re:Fossil versus Git by Culonavirus on Saturday September 23, 2023 @08:30AM (#37622691)
Not in gamedev where you can have hundreds of gigs of art assets (models, textures, audio...), but you still want to version them or even have people working on them at the same time (maps...). But that is a different can of worms entirely.
Re:Fossil versus Git by nightfader on Saturday September 23, 2023 @08:36AM (#37622724)
That's what object storage with versioning turned on is for e.g. GCS or S3
Re:Fossil versus Git by sausagefeet on Saturday September 23, 2023 @08:43AM (#37622748)
Why use git at all then? Just use an object store with versioning turned on.
Re:Fossil versus Git by nightfader on Saturday September 23, 2023 @08:47AM (#37622778)
Because git excels in relatively small size text files and patching and difficult. You can't binary blobs like jpegs, audio, video easily.
Re:Fossil versus Git by sausagefeet on Saturday September 23, 2023 @05:06PM (#37627558)
But that's my point: why can't a version control system be good for this as well? It's the same thing underneath. Why do we have to split these different use cases across different tools and hope a foreign key constraint holds?
Re:Fossil versus Git by Groxx on Sunday September 24, 2023 @01:08PM (#37634964)
We've got lots of disk-backup tools that handle this just fine, deduplicating blocks and compressing where they're able.
Re:Fossil versus Git by gchamonlive on Saturday September 23, 2023 @08:47AM (#37622780)
Although blob storage work well for versioning, you have to make heavy use of the underlying proprietary API to get these versions, and I am not quite sure you can do more complex operations, like diff and bisect between those versions the way you could with git.
Re:Fossil versus Git by mgaunard on Saturday September 23, 2023 @08:44AM (#37622755)
git is for when you want to track changes to part of a file.

In your scenario, you just want to track different versions of a file.

Re:Fossil versus Git by Groxx on Sunday September 24, 2023 @01:03PM (#37634899)
You can equally say that git is for when you want to track changes. And then it's a failing of git.

Besides, what's the difference? It's a file. The contents changed. Git doesn't say anything at all along the lines of "30% or more different means it's not a good fit for git".

Re:Fossil versus Git by mgaunard on Monday September 25, 2023 @03:55PM (#37649673)
You can certainly use git, but then the model used to apply patches from different branches doesn't work for binary blobs.

So all the things that git specializes in are actually things you don't want.

Re:Fossil versus Git by Groxx on Monday September 25, 2023 @05:41PM (#37651092)
[delayed]
Re:Fossil versus Git by firesteelrain on Saturday September 23, 2023 @08:49AM (#37622798)
Then use a binary repository like Artifactory which has LFS support. You can still version them in git- just don’t store them in there
Re:Fossil versus Git by otp209 on Saturday September 23, 2023 @12:46PM (#37624924)
> Then use a binary repository [...] You can still version them in git- just don’t store them in there

So git works, as long as you include things that aren't git to handle what it can't.

Stockholm syndrome really blinds people

Re:Fossil versus Git by firesteelrain on Saturday September 23, 2023 @05:51PM (#37627877)
Not sure if you were agreeing with me or not BUT I run into this often where people do not use the right tools and try to make one tool fit every CM scenario. SharePoint sucks but it has its place. Along with Artifactory/Nexus
Re:Fossil versus Git by dognotdog on Saturday September 23, 2023 @09:03AM (#37622887)
Indeed I have 3D assets in this case. Would this be done differently in an enterprise that has all kinds of tools to manage specialty workflows? Sure. Do I want to spend my days configuring and maintaining some binary blob / LFS storage system? No.

I’ve migrated a lot of projects from fossil to git eventually, but I dare say they never would have made it that far, had I started out with more friction, including fighting vcs tools.

Re:Fossil versus Git by Supermancho on Saturday September 23, 2023 @08:25AM (#37622667)
A workflow design problem?
Re:Fossil versus Git by thfuran on Saturday September 23, 2023 @10:18AM (#37623435)
That's a ridiculous claim. Can you really not think of a single situation in which it makes sense to keep track of big pieces of data alongside (or even instead of) source code? The fact that many VCS don't handle large binary data nicely doesn't mean there's never a reason to do so.
Re:Fossil versus Git by otp209 on Saturday September 23, 2023 @12:50PM (#37624960)
> Can you really not think of a single situation

It doesn't even matter if they can think of one; assuming your own use cases for software are everyone's is proceeding from false premises and is the sort of thing that leads to (and necessitates) "hacky workarounds" and eventually the adoption of better software we should've had in the first place.

Assume nothing about user's use cases. A VCS should not be imposing arbitrary limitations on the files it's indexing. It's like the old-school filesystems we (surprise, surprise) deprecated.

Re:Fossil versus Git by otp209 on Saturday September 23, 2023 @12:41PM (#37624886)
Your workflow and use cases aren't everyone's.