News for hackers. Stuff that matters.
Re:Git Large File Storage 1.0 by vvanders on Thursday October 01, 2015 @03:50PM (#10314160)
Without locking this is largely useless.

Usually large files are binary blobs(PSD, .ma, etc) and it becomes incredibly easy to blow away someone's work by not pulling before every file you edit(or two people edit at the same time).

As much as some people hate Perforce that's exactly what they are setup to do. Plus their binary syncing algorithms are top-notch. We used to regularly pull ~300GB art repo(for gamedev) in ~20 minutes.

Git is great for code but this seems like square peg, round hole to me.

Re:Git Large File Storage 1.0 by luckydude on Thursday October 01, 2015 @08:47PM (#10315894)
I've read the replies to vvanders and he's correct. With binaries you really want some sort of global locking (easy with a centralized system, hard with a distributed system).

I believe his (her?) point is that for a very large class of binaries there is just no upside in parallel development, one guy is going squash the other guy's work. You want to serialize those efforts.

We don't have global locks yet but we know how to do them, just waiting for the right sales prospect to "force" us to do them. I'm 90% sure we could add them to BK in about a week.

Re:Git Large File Storage 1.0 by icebraining on Thursday October 01, 2015 @05:42PM (#10314962)
Git annex solves this without locking nor losing any of the versions - the actual files get different names (based on hashes of the contents), which are referenced by a symlink tracked by git. If two people edit the same file - pointing the symlink at different filenames - you get a regular git merge conflict.
Re:Git Large File Storage 1.0 by vvanders on Thursday October 01, 2015 @06:21PM (#10315184)
That's the thing with binary assets, there is no merge path due to them being binary by nature.
Re:Git Large File Storage 1.0 by LukeShu on Thursday October 01, 2015 @06:44PM (#10315316)
No /automatic/ merge resolution. Obviously you have a tool that can open them (if you edited it in the first place), and you can use that to view the differences, and replay one set of changes. The fact that the SCM detected the conflict, and alerted you, allowing you to resolve it, is a solid improvement over not using an SCM. Further, automatic merge resolution isn't always possible with text-based assets either (and even when it is, it isn't always the correct option!).
Re:Git Large File Storage 1.0 by vvanders on Thursday October 01, 2015 @06:51PM (#10315368)
I don't think you follow. Programs like Photoshop, Maya and 3DSM don't merge.

Period.

Your only option in this case is to throw away someone's work and force them to redo the work on the file that you decided to keep.

Re:Git Large File Storage 1.0 by LukeShu on Thursday October 01, 2015 @07:36PM (#10315564)
Programs like gedit don't merge. Period.

Yet I can still use it to resolve a merge conflict.

In Photoshop you would do this by opening both images, and visually comparing them to see what's different, then copying the appropriate parts from one to another. Instead of just visually comparing them, you might combine them into one file as separate layers, and use a transform to see the difference. If the tool doesn't support that, use ImageMagick to generate a graphical diff (either the `compare` or `compose` commands), and then copy the relevant parts from one to another.

We have fancy tools to help us, but fundamentally, merging is a /human/ operation, that requires human judgment to see how multiple sets of changes can be made to coexist. And that doesn't require a tool (though it can certainly help).

Re:Git Large File Storage 1.0 by Xorlev on Thursday October 01, 2015 @08:12PM (#10315752)
> In Photoshop you would do this by opening both images, and visually comparing them to see what's different, then copying the appropriate parts from one to another.

Good luck.

The point the parent was trying to make is that the lock operation of SVN was quite convenient for preventing a dual-edit scenario of assets that aren't easy to merge like 3D meshes, scenes, PSDs, etc.. It's easy to sit in the ivory tower of text merge resolution given how easy it is in comparison. The atom of change in other tools are quite a bit less obvious. Sure you can diff a mesh, but merging usually just means redoing it or picking one or the other.

Re:Git Large File Storage 1.0 by vvanders on Thursday October 01, 2015 @08:18PM (#10315773)
Cool, how do you merge After Effects, Autocad, Cinema 4D, Unreal 3 Packages, Illustrator, Sketch, Blender, XSi Lightwave or any other of the production packages that I've seen used in shipping actual products? What happens if no one used layers in your Photoshop file and collapsed history to save performance?

There's a reason Pixar and most mid-large game dev studios use Perforce or a similar tech, it's because fundamentally you need locking if you're working with binary assets.

Re:Git Large File Storage 1.0 by drewm1980 on Friday October 02, 2015 @03:55AM (#10317254)
Your intelligent merge tool has access to the file history. If one user modifies a layer, and the other one squashes them down, in the merge you probably want to apply the changes in that order, even if it's out-of-order chronologically. If the file format has a full edit history baked in, great; even more info for the intelligent merge. Maybe the in-file history can even be kept in sync with the repository level history.

In the current ecosystem you probably need locking for your sanity, but some day software will suck less.

Re:Git Large File Storage 1.0 by brazzledazzle on Thursday October 01, 2015 @08:44PM (#10315884)
I can't imagine this is intended to compete head-to-head with something like Perforce. As you've pointed out it simply can't. But for a repo that's mostly code with binary assets that get updated occasionally it's probably a god-send.
Re:Git Large File Storage 1.0 by TeMPOraL on Thursday October 01, 2015 @11:05PM (#10316478)
> In Photoshop you would do this by opening both images, and visually comparing them to see what's different, (...)

Yeah, good luck with that. With software like Photoshop, not every change is obvious or easily visible. Maybe the other guy tweaked blending parameters of a layer, or reconfigured some effects layers. Or modified the document metadata. Or did thousands other things that are not immediately visible, or for which it is hard to determine the sequence changes should be reapplied.

Maybe you can manually merge the two files to some reasonably good approximation of the intended result, but you can't never be sure you didn't miss something important.

Merging tools for text files show enough information for you to know when you've seen every change made. You can't have that with complex binary formats used by graphics programs, mostly because those formats were never explicitly designed to support merging.

Re:Git Large File Storage 1.0 by LukeShu on Friday October 02, 2015 @10:41AM (#10318852)
> Yeah, good luck with that. With software like Photoshop, not every change is obvious or easily visible.

Well, there's the time-honored technique of "rapidly switch between windows that are zoomed to the same place." But, more rigorously, I mentioned a way to do this; there are tools that can do a diff of raster images--which is what you are making at the end of the day with Photoshop. Sure it can't tell you what blurring parameters someone changed, but you can see that the blur changed, then you can go look at the parameters.

> Or did thousands other things that are not immediately visible

The trickiness of that situation isn't unique to binary formats. It comes up with code too.

> Maybe you can manually merge the two files to some reasonably good approximation of the intended result, but you can't never be sure you didn't miss something important.

That's just as true with code as it is with other formats!

> because those formats were never explicitly designed to support merging.

Neither was text. We just ended up making some tools that were reasonably decent at it.

I've been there, I've done that. I've done the 3-way merge with Photoshop files, and resolved the conflicts with 2 different people working on an InDesign file, and broken down to running `diff` on hexdumps of PDF files. Resolving merges with things that don't have nice tools for it isn't fun.

But it's a /lie/ to claim that a conflict for binary formats is "game over, you're just going to steamroll someone's work, there is no path to merge resolution". It's not a fun path, but it's not game over. Which is all I was really trying to refute.

(aside: It's interesting to me that this chain of comments went from being upvoted last night to downvoted this morning.)

Re:Git Large File Storage 1.0 by TeMPOraL on Friday October 02, 2015 @11:34AM (#10319180)
> Well, there's the time-honored technique of "rapidly switch between windows that are zoomed to the same place." But, more rigorously, I mentioned a way to do this; there are tools that can do a diff of raster images--which is what you are making at the end of the day with Photoshop. Sure it can't tell you what blurring parameters someone changed, but you can see that the blur changed, then you can go look at the parameters.

I guess this could work with simple cases and if you accept less than pixel-perfect standard; I can see how this will fail when several people are working on a single file for long (because not everything that is important is visible to visual diff, at least you'd end up overwriting whatever scaffolding the other guy set himself up for his work), but at this point I'd be questioning the workflow that requires two or more people to work simultaneously on a single asset.

> The trickiness of that situation isn't unique to binary formats. It comes up with code too.

> That's just as true with code as it is with other formats!

Not really - text files don't contain any more data than you can see when you open them in your editor. With text, you see everything. When you open a 3D model or a PSD file, or even a Word document, what you see is just a tip of an iceberg.

> But it's a /lie/ to claim that a conflict for binary formats is "game over, you're just going to steamroll someone's work, there is no path to merge resolution". It's not a fun path, but it's not game over. Which is all I was really trying to refute.

I can agree with that. It's not impossible to do such merges; worst case scenario, one will end up praying to a hex editor like you say you did. It can even be fun sometimes. I guess what 'vvanders was arguing about is practicality - you can do it if you're willing to invest the time, but it's much better to not have to do it at all.

> (aside: It's interesting to me that this chain of comments went from being upvoted last night to downvoted this morning.)

    HN moves in a mysterious way
    its voting to perform;
    A reader questions his comment's downvotes,
    And thus ensues shitstorm.
That is to say, sometimes it's so random and fluctuating that personally, I stopped caring. If I get downvotes I usually a) already know I deserve them for being an ass and/or factually incorrect, b) have someone tell me why I deserve them, or c) assume they're random fluctiations and not worth getting upset about. I think we're dealing with type c) now.
Re:Git Large File Storage 1.0 by theseoafs on Friday October 02, 2015 @02:58PM (#10320521)
So your suggestion for merging two 3D models that were made in Blender is to use ImageMagick compare to see what's different and then copy the differences from one Blender file to the other?
Re:Git Large File Storage 1.0 by drewm1980 on Friday October 02, 2015 @03:42AM (#10317214)
Is that really inherent to the data, or a case of useful merge plug-ins just not having been written yet for the proprietary file formats of big closed source apps?

Source code doesn't ~really merge all that well either; there's just been a big community of software developers collaboratively fixing up their tools for collaboration.

Of course we have it easy cause the tools we're using are ~made of the same stuff we work with daily. No amount of photoshop filter experience will enable you to write a program that intelligently merges two photoshop files.

Re:Git Large File Storage 1.0 by jayd16 on Thursday October 01, 2015 @08:36PM (#10315857)
Except perforce has been moving towards streams to compete with git's more desirable workflow, and you can't lock across streams anyway.
Re:Git Large File Storage 1.0 by sytse on Friday October 02, 2015 @03:40AM (#10317209)
After we add support for git lfs we plan to add web ui locking for files, this will allow you to lock files when browsing them and prevent others from uploading them
Re:Git Large File Storage 1.0 by alkonaut on Thursday October 01, 2015 @05:15PM (#10314782)
Won't a central storage for the large files alao make it straightforward to add locking functionality in a future version of git-lfs, or as an add-in? I agree it sure looks like an omission to have a vcs that is centralized and aimed at binary data, without having any locking functionality.
Re:Git Large File Storage 1.0 by vvanders on Thursday October 01, 2015 @06:21PM (#10315190)
Yeah, but that's assuming a whole lot of infrastructure around marking files read-only.

I also haven't seen what the pull performance is like, P4 is a pretty known quantity(with caching as well).

Re:Git Large File Storage 1.0 by alkonaut on Friday October 02, 2015 @02:33AM (#10317039)
I don't even necessarily need my client to mark files as read only on my local machine. A system that just lets me query who has the lock on a certain file, and lets me get the lock on it if noone has, is miles ahead of shouting/email.

A remote-only locking system should be pretty easy to implement, e.g by just throwing in "filename.userid.lock" into the filesystem next to the file in question.

Re:Git Large File Storage 1.0 by sytse on Friday October 02, 2015 @03:43AM (#10317219)
We plan to add locking to the GitLab web ui at some point. At least this will prevent two people from locking the same file.
Re:Git Large File Storage 1.0 by bmurphy1976 on Thursday October 01, 2015 @08:39PM (#10315866)
Translation: because it is not useful for you it's not useful for anybody else.

That's nonsense of course.

There are a lot of use cases where this would be very helpful without locking (i.e. jar/dlls).

This is useful now, locking can come later. We don't have to solve every conceivable problem all at once. More progress is made in small incremental steps than big bang leaps.