LackOfGravitas
LackOfGravitas on Hacker News
Interesting. I'm always skeptical of blog posts companies make about themselves, but I can definitely see how block-level deduplication could be a huge help.
I wonder how it scales with the numbers of blocks (e.g. with 10 TB of data)? I would guess that it would really help with the amount of data stored but then slow way down as more data gets added due to the extra overhead of tracking and deduping.
Also, does it work across repositories?
What specifically is meant by "Modern Development Experience"? I see github eliminating a lot of useful development pain points with its tools for repository collaboration and integration, which is what I associate with the current trend with repositories, but this post is focusing just on having data with the code which is an interesting assumption...
Fair. I do have an archive folder in the home directory. Also, probably 70% of the space is in a large workspace/ folder that has different programming projects or open source tools in it. There are also a lot of art and graphics files, some raw and some the result of processing pipelines. Much of it is largely static, but not quite in the "Write Once, Keep Forever" category.
I definitely admit that all these could be on other drives, but in this case I have found it easier to just have everything together and make sure folders are organized intelligently. At least in this case, the overhead of having separate backup procedures for different types of data is more than the marginal overhead of simply snapshotting everything.
That said, I do have a 8 TB photography folder that isn't part of this snapshot routine.
I think the only size limitations would be the size of the data store -- my 1.2 TB home folder with 1.2 million or so files becomes around 400GB data store on my NAS and around 200MB for the actual repo. I know that git slows down with number of files, but I haven't noticed any issues yet.
Currently, it takes probably 20-30 minutes to run, which does include all the data deduplication. It was a bit more for the first round, but still under an hour. On my home folder, which is about 1.2 TB and I think around 1.4 million files, the git repository itself is a few hundred MB and the local data store, on a zSTD compressed ZFS data pool on my True NAS server, is around 400 GB. The data deduplication pretty much just tracks with new content, so it works perfectly here for snapshots.
Posted by
LackOfGravitas on Thursday January 04, 2024 @02:43PM
from the show-hn dept.
Is it easy to try this out on an existing repo and then uninstall it and go back to using git LFS if I don't like it? Can this be done without messing up my repo?
Makes sense! I'll try it out.
Interesting idea, and I can see the potential. However, other tools I've seen that try to "simplify" git end up breaking some desirable aspect of the git workflow, making it far less useful to me than just using the git workflow I'm comfortable with directly. Could you explain a bit more about what mental model I should have in mind when using your tool and what limitations I'd end up running in to?