News for hackers. Stuff that matters.
Re:Why Google Stores Billions of Lines of Code in a Single R... by jayd16 on Saturday January 11, 2020 @12:22PM (#22021014)
Monorepos are cool but I hate perforce with a passion. How can we use git?

A lot of work is going into making git scale better which is nice but there will always be small open source repos. You could mirror them but it seems like it would be nice to not have to do that.

Is there any research going into figuring out how to keep many repos in sync with a single commit? Maybe you could use submodules across your company...have a single master repo that essentially just keeps track of working version sets. That might work but it sounds torturous. Is anything on the horizon to solve this problem with git?

Re:Why Google Stores Billions of Lines of Code in a Single R... by alexhutcheson on Saturday January 11, 2020 @01:15PM (#22021353)
Google has published the git-repo tool[1][2], which is helpful for working across multiple git repos. Even with the tool there are significant pain points, though.

In the longer term, I'm hopeful that VFS for Git[3] (from Microsoft) and/or Mononoke[4] (from Facebook) will reach a level of maturity that makes it easy to host large monorepos using open-source distributed version control systems.

In the short-term most smaller organizations aren't going to hit the technical scaling limits of Git anyway. Most organizations could put all their projects in a single Git repo and not have any significant performance issues. Organizational issues like commit ACLs and code review enforcement are a separate issue, and often require additional tooling.

[1] https://gerrit.googlesource.com/git-repo/

[2] https://source.android.com/setup/develop/repo

[3] https://vfsforgit.org/

[4] https://github.com/facebookexperimental/mononoke

Re:Why Google Stores Billions of Lines of Code in a Single R... by dmoy on Saturday January 11, 2020 @02:52PM (#22021882)
The short version is that you can't do this with git, because it doesn't scale.

See Facebook's discussion with git devs here: https://news.ycombinator.com/item?id=3548824

The long version is that many are trying. MS is making some headway last I checked.

Re:Why Google Stores Billions of Lines of Code in a Single R... by kinkrtyavimoodh on Saturday January 11, 2020 @12:45PM (#22021175)
Git is great (I love it) but Google anyway uses a fairly modded form of Perforce and the data model etc. is much simpler. Especially with use of IDEs it makes it so that you don't have to keep thinking about your VCS, unlike git, where GUIs come in the way.
Re:Why Google Stores Billions of Lines of Code in a Single R... by jayd16 on Saturday January 11, 2020 @12:53PM (#22021222)
Ah sorry. I don't want to speak to what Google has made. I've never used it. Compared to the public version of Perforce, git has a much simpler model. Streams, view specs, branch mappings, p4admin tooling, a myriad of Perforce plugins of varying quality you're expected to use, and an official gui that is embarrassingly bad make Perforce much more complex in my mind. (Although it's doing more things so it's not without reason...)

The one thing p4 does have is file level versioning. It's trivial to go back and pick single versions of files. Artists need that feature and most git clients hide it away or don't support it at all.

Re:Why Google Stores Billions of Lines of Code in a Single R... by forwhomst on Saturday January 11, 2020 @12:36PM (#22021121)
Why do you dislike perforce? I’ve used them all and git is the worst, in my opinion, of the ones that aren’t dead and buried (like cvs). Perforce is my fave.

Internal research at google suggested that ‘git5’ users were objectively less productive than average, but there’s some friction around ‘git5’ so that doesn’t necessarily implicate git itself.

Re:Why Google Stores Billions of Lines of Code in a Single R... by rak00n on Saturday January 11, 2020 @01:21PM (#22021386)
You have to be bluffing. I never heard of such internal research. How is even productivity measured in a context like this?
Re:Why Google Stores Billions of Lines of Code in a Single R... by joshuamorton on Saturday January 11, 2020 @03:01PM (#22021926)
Google has an entire engineering productivity research team. They do research on tooling and such all the time.
Re:Why Google Stores Billions of Lines of Code in a Single R... by rak00n on Saturday January 11, 2020 @05:18PM (#22022750)
I wasn’t doubting on existence of a research team like that. I was doubting on the research mentioned. You cannot just say people who use a particular tool (like git or git5) are less productive. That’s just absurd.
Re:Why Google Stores Billions of Lines of Code in a Single R... by joshuamorton on Saturday January 11, 2020 @05:40PM (#22022886)
I don't think the research is public, because it's all entirely about internal tools, but yes the conclusion was that the got interface for perforce led to less efficient development across basically any metric you could look at (LoC, commit count, self-reported efficacy, etc.). Note that if you judge developers based on commit count is a bad metric, but comparing LoC throughout of developers who are evaluated on other metrics isn't problematic.
Re:Why Google Stores Billions of Lines of Code in a Single R... by rak00n on Sunday January 12, 2020 @02:25AM (#22025016)
I just searched in moma with variations of “git5 productivity” and no such study showed up. Neither LoC nor commit count is a good metric to represent a developer’s productivity.

In my personal opinion, using a git/hg like interface makes it lot easier to work on a complicated CL because you can maintain internal local branches and you can easily revert your incremental changes. That’s not at all possible in perforce. I just can’t see how git/hg interface can make anyone less productive.

Re:Why Google Stores Billions of Lines of Code in a Single R... by joshuamorton on Sunday January 12, 2020 @02:42AM (#22025061)
> Neither LoC nor commit count is a good metric to represent a developer’s productivity.

They're not a good metric to judge a developer on. But when you can do a large controlled study (or even before/after with the same developer), without the developer knowing they're being watched, it's a good metric.

Re:Why Google Stores Billions of Lines of Code in a Single R... by jayd16 on Saturday January 11, 2020 @01:01PM (#22021279)
I mostly dislike the general bugginess, the need to use plugins of varying quality in all your tools, the centralized design and/or poor dvcs support, and the strange feature dead zones you get. As far as I can tell (and I opened a ticket but maybe I pulled a bad support tech) it's impossible to see what CLs were merged into a branch, only what still needs to be merged.

The general low quality of a lot of the Perforce tools has me scratching my head every day. How hard is it to make a log window that scrolls to the bottom? The p4proxy doesn't have a max size, you need to manually delete files on the machine to free space!?

Re:Why Google Stores Billions of Lines of Code in a Single R... by forwhomst on Saturday January 11, 2020 @01:44PM (#22021532)
You can definitely inspect the changes that have been integrated into a perforce branch with filelog and changes, it's not worth pointing out exactly how, you can find examples online (much like git). And if you don't like how p4proxy behaves when it runs out of space, just wait until you experience git with a full disk.
Re:Why Google Stores Billions of Lines of Code in a Single R... by jayd16 on Saturday January 11, 2020 @04:31PM (#22022464)
Git works better. You can prune git or delete lfs files remotely. P4 proxy stores every file revision it touches and does no book keeping. Revision count is not honored and obliterate doesn't clean it up. P4 proxy will grow unbounded and take up much more space than your master repo. The only way to clean it up is manually deleting raw files.

As for a single command to see already merged/copied CLs between two streams, please enlighten me. I want the unsquashed CLs. It would be really useful for generating patch notes but because p4 does version tracking at the file level and you can merge some files from a commit and not others etc, it's not trivial at all.