News for hackers. Stuff that matters.
Re:Torvalds: git vs CVS by ableal on Friday April 16, 2010 @05:40PM (#1272009)
Exercise: what are the downsides of the current generation of DVCS ?

(Assume you may be something other than a coder that carries all files in a laptop.)

Re:Torvalds: git vs CVS by nuclear_eclipse on Friday April 16, 2010 @05:57PM (#1272062)
Poor or inefficient support for large binary assets, such as compiled objects, images, or video. Especially in the context of video game development, binary assets that constantly change are highly painful to deal with in Git or Mercurial.
Re:Torvalds: git vs CVS by lnguyen on Friday April 16, 2010 @06:34PM (#1272149)
It's not limited to DVCS or open source. Commercial tools such as ClearCase, AllFusion/Harvest, and Dimensions still handle large binary assets poorly.

I don't think you'll find a satisfactory tool that has as its primary use the management of texts and deltas of texts. You'd probably need to use a combination of tools which has its own set of frustrations.

Re:Torvalds: git vs CVS by smallblacksun on Friday April 16, 2010 @07:05PM (#1272209)
Perforce works well with large binaries and text (costs a fortune, though).
Re:Torvalds: git vs CVS by sandGorgon on Friday April 16, 2010 @10:37PM (#1272489)
Bitkeeper (the SCM that git was "inspired" from), claims good support for large binary assets

BAM solves this problem with a hybrid approach. BAM adds the concept of one or more BAM servers. Each BAM server contain all BAM data, but clones of the server (work spaces) contain only that data that the user requests. Typically this is the most recent version of each binary but in some cases it may even be less than that.

BAM has been successfully deployed in the game development space with great success. Game developers continue to enjoy the benefits of distributed work flow without the penalty of carrying all versions in every workspace.

Re:Torvalds: git vs CVS by naner on Friday April 16, 2010 @06:08PM (#1272095)
How is it possible that nobody has dealt with this problem yet?
Re:Torvalds: git vs CVS by pmjordan on Friday April 16, 2010 @06:13PM (#1272109)
Active open source development rarely meets "AAA" game development.
Re:Torvalds: git vs CVS by sshumaker on Friday April 16, 2010 @07:45PM (#1272272)
One could argue that the traditional source control model (where everyone has a copy on their own machine) isn't a great way to deal with huge binary assets. I've worked on game projects where the binary churn was on the order of gigabytes a day. Even though everyone was physically co-located, when everyone arrived at 10am and started to update to head, a hundred computers all trying to download the days worth of 2 gigs of assets over the network didn't make for a pretty picture.

You can try to cronjob the updates, staggered around 4am. Or you can realize that the majority of people are only modifying a subset of these assets, and roll your own link-based asset manager for large binaries.

Re:Torvalds: git vs CVS by pmjordan on Saturday April 17, 2010 @06:11AM (#1272864)
Yes, absolutely, you wouldn't want everyone to keep pulling all data, git-style. The programmers usually only need the assets in the game-ready format, not the source models, textures, etc. which are typically much larger, and only needed by the artists/level designers working with them.
Re:Torvalds: git vs CVS by blasdel on Saturday April 17, 2010 @03:53AM (#1272779)
Because anyone cleverer than a doorknob uses the right tool for the job: rsync. They're almost always artifacts, not assets.

You handle versioning either through filesystem snapshots, or by just using different paths for each version.

Re:Torvalds: git vs CVS by ableal on Saturday April 17, 2010 @04:30AM (#1272812)
So, git/hg/etc are doorknobs ?

Of course not, it's not their problem. What annoys me is that the "You must use DVCS" memo is being sent around uncritically, and anyone who does not fit the use case is left out.

I think that people who deal with images/spreadsheets/data would also benefit from a good VCS tool. If the coders, having scratched their itch, did not declare the problem solved and move on ...

P.S. thanks for the polite answer. So, there is either no problem or no solution ... I think we're done here.

Re:Torvalds: git vs CVS by blasdel on Saturday April 17, 2010 @04:45AM (#1272823)
git/hg/etc are not doorknobs. My simile was comparing you, and people like you, to doorknobs. http://www.google.com/search?q=dumber+than+a+doorknob

No version control system, centralized or distributed, actually handles blobs better than just treating them as artifacts and using rsync.

The only way to actually solve the problem is to make the data not be blobs anymore, where the diff/merge/serialization code all understands at least the structural container format and can render it usefully. There's never going to be a general purpose tool that does that outside of a live-in smalltalk image (or similar).

The best you're going to get is tools vertically-integrated with the application, and all the ones I've seen to date (MS Office, Adobe Version Cue, etc.) all do a terrible job of even doing diffs/RCS, much less actually implementing a real VCS.

Re:Torvalds: git vs CVS by stevelosh on Saturday April 17, 2010 @10:30AM (#1273071)
Greg Ward and some others are working on it with the bfiles extension for Mercurial. It's a work in progress but at least it's getting the ball rolling: http://mercurial.selenic.com/wiki/BfilesExtension
Re:Torvalds: git vs CVS by ableal on Sunday April 18, 2010 @02:04AM (#1274452)
Thank you very much. I liked what I read in the design doc (http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt), which seems to be trying to automatically do the best thing possible under the hg constraints.

It's interesting that the names are "Distributed Version Control Systems", or "Revision Control", but the ghost of the word "source" (meaning code text, in 1972's SCCS) is still read into it.

Surprisingly, when DVCS are shouted as the bees' knees for everybody, people who have images, documents and data may think they were included. Apparently not, those benighted heathens should use rsync or whatever manually, and not soil the "source control" systems with their binaries. Or go find not-included plug-ins, if they can.

(Those people are apparently too dumb to understand that a binary "artifact" produced with an image editor, spreadsheet, or data analysis tool is fundamentally different from a blessed code file written with a text editor, and thus is not entitled to be under real version control, or have a DCVS tool automatically do something smart about it.)

The Perforce guys must be laughing their asses off.

Re:Torvalds: git vs CVS by ableal on Friday April 16, 2010 @06:23PM (#1272122)
I have been trying to point it out [1], without much luck. Today I got in early by chance, and decided on the Socratic approach ;-)

[1] http://news.ycombinator.com/item?id=1201559, http://news.ycombinator.com/item?id=1219082, http://news.ycombinator.com/item?id=1222905, http://news.ycombinator.com/item?id=1242374

Re:Torvalds: git vs CVS by prog on Friday April 16, 2010 @10:31PM (#1272482)
This is my pet peeve with all DVCSes. I tried this with bzr, hg and git and all fail. Not sure if any other DVCSes handle it.

I don't see it happening anytime soon as long as it justified using "its _source_ control". There are programming activities that require large binary files to be versioned along with source and this is a clearly a limitation of DVCSes IMO.

Apart from that I am quite happy moving to bzr from svn. Life is much easier now.

Re:Torvalds: git vs CVS by ajb on Saturday April 17, 2010 @07:34AM (#1272931)
Apparently vesta (http://www.vestasys.org) can handle large binaries (it's from the EDA world, so this is plausible). It's open source, but the extent to which it's distributed is debatable - you need a server, but anyone can install a server and they form a distributed network like a dvcs.
Re:Torvalds: git vs CVS by ableal on Sunday April 18, 2010 @12:15PM (#1275009)
Thanks, looks interesting, and it came from DEC via Compaq. It's billed as a Configuration Management System (i.e. managing builds). They do talk of managing tens of GB of derived data, which is also important.

The hardware part seems limited: "the Alpha group ... hardware description language files into Vesta's source code control", so not schematics or layout binaries as 'source'.

That site seems to be resting since 2006, but there is some activity at http://sourceforge.net/projects/vesta/ (releases in 2009, mailing lists with 2010 messages).

Re:Torvalds: git vs CVS by ajb on Sunday April 18, 2010 @02:42PM (#1275243)
Last time I looked, the IRC channel was somewhat more active than the list.
Re:Torvalds: git vs CVS by sshumaker on Friday April 16, 2010 @07:55PM (#1272281)
With binary files, you really want an explicit checkout (exclusive lock) system - since there's typically no way to merge them. This makes the commit/merge style SCSes a bad fit for binary assets (and partially explains the popularity of Perforce, which is a lock-based SCS).
Re:Torvalds: git vs CVS by dalore on Friday April 16, 2010 @08:56PM (#1272364)
That and Perforce is a commercial company that provides support with their product. By default perforce doesn't lock, but it knows when a user has a file checked out for "edit".

By default all files are set read only. This was a big downside when you had no internet connection.

Re:Torvalds: git vs CVS by koenigdavidmj on Saturday April 17, 2010 @03:17AM (#1272750)
Yeah, but 'chmod u+w filename' and you're back in business. The GUI has a "resolve offline work" feature that will find everything.

Of course, if you don't mark it writable (and just use :w! in Vim, for instance), then a 'p4 sync' may eat your work if anyone's changed the file.

Re:Torvalds: git vs CVS by dalore on Monday April 26, 2010 @06:43AM (#1294553)
Ahh the resolve offline work feature must be new. It wasn't around when I was using perforce.
Re:Torvalds: git vs CVS by koenigdavidmj on Monday April 26, 2010 @10:41PM (#1297026)
Sadly, it has no direct CLI equivalent. You may find http://p4delta.sourceforge.net/ useful, however; it's like `svn up` for Perforce.
Re:Torvalds: git vs CVS by chrisbolt on Friday April 16, 2010 @05:46PM (#1272028)
Inability to check out a subset of a repository.
Re:Torvalds: git vs CVS by snprbob86 on Friday April 16, 2010 @06:37PM (#1272160)
There are two good reasons to checkout only a subset of the repository:

1) Large binary or data files -- For this, there are various solutions of varying hackiness.

2) Your project has grown into several sub-projects -- The best solution here is to clone the repository to new names and then refactor and trim the respective project trees with a lot of deleting. You'll thank yourself later for improving your architecture and there will no longer be a partial check-out problem.

The real issue for both of these, is handling unversioned or independently versioned bits with respect to all the other bits. You might want some unversioned binary blobs, but you might also want some versioned giant csv files without having to download all the old versions all the time. Source code has one versioning strategy, data files another, photoshop files yet another. No version control system today really lets you pick which versioning strategy to use for which files. And none of the existing "sub-module" type extensions are really great either.

Re:Torvalds: git vs CVS by alextp on Friday April 16, 2010 @08:48PM (#1272355)
Actually, the worst problem with submodules in modern DVCSs is handling dependencies. If project A depends on project B and project B and the API exposed by project B varies over time it is pretty easy to keep both in sync and working. Except, when someone needs to check out an older version of A they need to figure out which version of B to check out (and it might not be as easy as checking out the latest B in the date that that version of A was committed). Git allows you to mark a specific revision of a submodule, but then updating the whole project is no longer a single command, and there are some problems with not pushing everything at the same time.
Re:Torvalds: git vs CVS by sandGorgon on Friday April 16, 2010 @10:33PM (#1272483)
http://mercurial.selenic.com/bts/issue105

Two of the features that made it difficult to 'sell' Hg for the main SCM of FreeBSD were the lack of Partial checkouts and Partial history

Re:Torvalds: git vs CVS by viraptor on Friday April 16, 2010 @08:46PM (#1272353)
3) You have a small patch and want to update it to the HEAD version before submitting. You worked with the source you obtained in some other way (from a source package for example). Now you have to wait 10+ minutes on a large project, downloading all the changes since 1970, just to change the newest version of a single file.

4) The project has grown into several sub-projects - but you don't control that repository.

Re:Torvalds: git vs CVS by cma on Saturday April 17, 2010 @01:45AM (#1272686)
re #3:

"shallow repository A shallow repository has an incomplete history some of whose commits have parents cauterized away (in other words, git is told to pretend that these commits do not have the parents, even though they are recorded in the commit object). This is sometimes useful when you are interested only in the recent history of a project even though the real history recorded in the upstream is much larger. A shallow repository is created by giving the —depth option to git-clone(1), and its history can be later deepened with git-fetch(1)."

There are still issues; you can't commit for instance, but you can update to HEAD to update your patch, with a depth of 1 you only get the most recent changes.

Re:Torvalds: git vs CVS by dasil003 on Friday April 16, 2010 @09:08PM (#1272379)
I agree this is a definite weakness compared to svn, but the problem is that svn gains this facility by having no concept of a project. Let me repeat that: in svn it is technically impossible to identify the project tree.

The problem with that, of course, is that merging only makes sense if you are merging branches that occur at the same tree-level in a project. This is why subversion merge tracking is so buggy and half-baked... because any given directory could be a project tree, a subtree, a branch, or a tag, and you could merge any of them. Sure if stick to certain conventions it works pretty well, but technically the whole implementation is a minefield, which is why svn will never be as robust as other systems that don't make this mistake.

Even though git's submodules and subtrees leave a lot to be desired, and have significant room for improvement, they will never be as convenient for partial checkouts, because the requirements for svn-style partial checkouts require crippling the entire system.

Re:Torvalds: git vs CVS by prog on Friday April 16, 2010 @10:27PM (#1272475)
bzr has support for filtered views[1] if thats what you mean.

[1] http://doc.bazaar.canonical.com/latest/en/user-guide/filtere...

Re:Torvalds: git vs CVS by dryicerx on Friday April 16, 2010 @06:21PM (#1272119)
For this reason, I am loving svn-git
Re:Torvalds: git vs CVS by koenbok on Friday April 16, 2010 @06:00PM (#1272077)
- When you work with large binary files that don't diff well (psd), it's a pain to have the entire repository history local.

- Hard to explain to even fairly technical designers/copywriters.

- With all that easy branching, I sometimes just have a hard time getting a decent version of the codebase together from everyone.

Re:Torvalds: git vs CVS by dustingetz on Friday April 16, 2010 @06:18PM (#1272114)
i used SVN to track large binaries and it was morbidly slow. maybe i was doing it wrong. i thought the solution was to keep binary assets under a separate version control better suited for files that don't diff; a nightly snapshotted filesystem seems just fine.
Re:Torvalds: git vs CVS by ableal on Friday April 16, 2010 @06:26PM (#1272127)
Seems that's why Perforce is in business (e.g. http://news.ycombinator.com/item?id=1242374 )
Re:Torvalds: git vs CVS by th on Friday April 16, 2010 @07:10PM (#1272221)
Complexity (in terms of the operations that need to be performed regularly). At least with Git, simplicity seems to be traded in favor of flexibility.
Re:Torvalds: git vs CVS by prog on Friday April 16, 2010 @10:25PM (#1272473)
bzr, hg and darcs are not too complex to use.
Re:Torvalds: git vs CVS by ionfish on Friday April 16, 2010 @06:00PM (#1272072)
No way to move files between repositories and retain their history. Luke Palmer talked about this a little here:

http://lukepalmer.wordpress.com/2008/11/12/sketch-of-udon-ve...

Re:Torvalds: git vs CVS by blasdel on Friday April 16, 2010 @09:00PM (#1272369)
It is absolutely possible to do this in git. It helps if you forget about "files" as something to "move".

I did it in the project I'm working on now! It originally had one main repository, and another for an extension that was used as a git submodule in the main one. Active development proceeded in both, with shitloads of commits in the main one just to update the reference to the submodule.

I decided this was retarded, so I took a clone of the extension repository, used filter-branch to rewrite it's entire history so that all the paths were always prefixed with "vendor/extensions/project_name/", and pushed it as an unrelated branch into the main repository.

Then in the main repository I made a new branch, removed the old submodule junk there, did a nice clean merge that melded all the commits from the extension branch going back in time, and made that the new master branch.

Re:Torvalds: git vs CVS by windsurfer on Friday April 16, 2010 @07:06PM (#1272213)

  git filter-branch --subdirectory-filter foodir -- --all
Re:Torvalds: git vs CVS by _delirium on Friday April 16, 2010 @07:29PM (#1272250)
Well, a big one is probably the same one: politics. Instead of politics over commit privileges and commit rules, you have politics over who controls mainline, who controls which staging areas, what their rules are for pulling or accepting pushed patches, etc.
Re:Torvalds: git vs CVS by bdonlan on Friday April 16, 2010 @07:47PM (#1272273)
The point is with centralized systems, only committers can do things such as branching or preparing commits. With a DVCS, everyone gets the same tools, so the rights granted by having the patch accepted are simply having it merged, nothing more or less.
Re:Torvalds: git vs CVS by _delirium on Saturday April 17, 2010 @03:59AM (#1272790)
That's true, and I think it is an improvement when it comes to the ability to do local forks. But most of the big fights I've seen in CVS/SVN-based projects ultimately come down to who controls mainline, and a DVCS doesn't solve that problem at all. If anything, it makes it easier to put even more politics on top of it, where instead of having a fight over getting into the single repository, you now have to navigate whole cascading set of repositories (e.g. Linus doesn't generally pull directly from you, so you have to deal with sub-gatekeepers before you get to the main gatekeeper).

I suspect Linus likes the approach not because it has no politics, but because it encourages the politics that match the way he'd like to manage the Linux kernel development, with this style of hierarchical, cascading commit approval, which is pretty hard to set up in CVS (though Mozilla's sort of grafted it on via their Bugzilla, which keeps track of cascading approvals of patches based on who owns which areas and sub-areas). Not that that's necessarily a bad thing, it's just different (and for a project as large as Linux, probably necessary).

Re:Torvalds: git vs CVS by yummyfajitas on Friday April 16, 2010 @05:54PM (#1272051)
Lack of a good way to edit recent history.

Suppose I sit down for a collaborative session with someone. Over the course of a few hours, we might generate 10-100 commits between the two of us, and several branches, all to implement a single feature. I.e., I commit, say "hey, pull from me, you had an off by one a few minutes ago."

At the end of the session, we only want to generate a single commit to the public repositories.

Mercurial queues are a tolerable way to handle this, but they aren't great.

Re:Torvalds: git vs CVS by nuclear_eclipse on Friday April 16, 2010 @05:59PM (#1272068)
Git makes this easy. It supports "rebasing" commit trees, and for what you want, a "squashed" rebase is absolutely trivial to do.
Re:Torvalds: git vs CVS by vtail on Friday April 16, 2010 @06:02PM (#1272082)
Re:Torvalds: git vs CVS by dalore on Friday April 16, 2010 @08:59PM (#1272366)
I do this all the time with git. I create a branch, do all my incremental commits, get it all working. Then I merge it into the mainline as one squashed commit. There a few guides on how to do it with the command line but I'll admit I've gotten lazy and now just use smartgit which lets me do it easily without having to remember the commands.
Re:Torvalds: git vs CVS by prog on Friday April 16, 2010 @10:37PM (#1272488)
git already has this in core and bzr has a rebase plugin[1]. I think this problem has been solved for quite some time now :)

[1] http://doc.bazaar.canonical.com/plugins/en/rebase-plugin.htm...

Re:Torvalds: git vs CVS by cool-RR on Saturday April 17, 2010 @10:25AM (#1273064)
I would want something pretty superficial: An ability to organize branches in folders.
Re:Torvalds: git vs CVS by maw on Saturday April 17, 2010 @10:37AM (#1273082)
bzr can do that with its shared repos. It's not too hard to fake it with git, either. (This assumes you know how to use git, which without a doubt is hard, despite what its apologists may tell you.)
Re:Torvalds: git vs CVS by jedediah on Friday April 16, 2010 @11:21PM (#1272550)
Terrible submodule support.