News for hackers. Stuff that matters.
Re:Microsoft and GitHub team up to take Git virtual file sys... by perfmode on Friday November 17, 2017 @04:33PM (#15726144)
IMHO it is a major failure that git doesn’t scale for org-wide mono repos. Software development process should not be dictated by limitations in the VCS. It surprises me that others are so willing to settle.
Re:Microsoft and GitHub team up to take Git virtual file sys... by jlgaddis on Friday November 17, 2017 @05:12PM (#15726425)
> IMHO it is a major failure that git doesn’t scale for org-wide mono repos.

git was (initially) written by Torvalds to serve his own purpose -- he needed a new VCS for Linux. Let's say that it "doesn't scale for org-wide mono repos". Unless that was a design goal (and I'm pretty sure it wasn't), how is that a "failure"? It apparently works just fine for the kernel developers.

Yet because it didn't work well for Microsoft's 300 GB worth of tens millions of lines of code it is somehow a "failure"?

Apparently Microsoft's own TFS is a failure too, then, because it seems that it wasn't up to the task either.

Re:Microsoft and GitHub team up to take Git virtual file sys... by ethomson on Saturday November 18, 2017 @07:55AM (#15728777)
PM at Microsoft here. TFVC is completely different, though.

(To quickly clarify nomenclature: TFVC is "Team Foundation Version Control", the name of the centralized version control system in TFS and VSTS, which are our on-premises and cloud hosted development platforms, respectively. They both support both TFVC and Git - Git with GVFS, and they do more than just version control, hence the naming clarification. Apologies for all the three letter acronyms.)

Anyway, TFVC is totally different and a centralized version control system with all the workflows that go along with it like expensive branching. The desire to move to Git opens up all the awesome workflows that Git offers.

The impetus here was to standardize the entire company on a modern set of development tools - one engineering system across the company - hosted in Visual Studio Team Services. We could have moved them to TFVC, it's remarkably similar to its predecessor, "Source Depot" which is a Microsoft internal tool that the Windows team was using.

But that's a lateral move. There aren't really any benefits to TFVC over Source Depot except that we sell and support one to end users and don't with the other. So that has some nice organizational benefits but not enough to warrant moving everybody and the build/release farms over.

But moving to Git unlocks all sorts of new workflows - lightweight branching, pull requests, all that good stuff that everybody who uses Git is totally used to.

Anyway, that's the background on Git vs TFVC. And, no, I totally agree that Git not supporting giant monorepos is not a failing, per se. But it is a limitation, and thankfully one that we're working to help overcome.

Re:Microsoft and GitHub team up to take Git virtual file sys... by merb on Saturday November 18, 2017 @06:11PM (#15731197)
well what would be cool if TFS could be installed on Linux and use a different database than MSSQL (PostgreSQL, MySQL or MSSQL) (I mean search uses elasticsearch anyway...). Even better if it would be more close to the online service. And way better if it came with something like gitlab-runner.
Re:Microsoft and GitHub team up to take Git virtual file sys... by ethomson on Saturday November 18, 2017 @07:35PM (#15731549)
Can you clarify what you mean by "more close to the online service"? We update TFS with the bits from VSTS pretty regularly; are you looking for a faster cadence or am I misunderstanding what you're saying?
Re:Microsoft and GitHub team up to take Git virtual file sys... by merb on Sunday November 19, 2017 @04:21AM (#15733032)
well some views are looking more clean on the online version. i.e. the project overview page.

currently I'm evaluating git hosting on-premise and another downside is, is that git tfs takes a lot of horse power (this looks unsuitable for a small team)

Re:Microsoft and GitHub team up to take Git virtual file sys... by ryanisnan on Friday November 17, 2017 @07:48PM (#15727150)
There's also the popularity factor... not many care(d) about TFS. I would refrain from calling most pieces of technology a "failure", git especially.

When your screwdriver does a bad job at pounding nails, you're doing it wrong.

Re:Microsoft and GitHub team up to take Git virtual file sys... by jonhohle on Friday November 17, 2017 @05:08PM (#15726406)
Is there a VCS that does?

When I was at Amazon, Perforce couldn't scale (it had already been partitioned into a few depots, but those had reached their reasonable capacity). Amazon moved to a per "module" git-repo system. It was convenient to push across multiple repos simultaneously, but it wasn't a deal breaker when we lost it (I never heard of any workflows being completely unworkable following the transition).

Re:Microsoft and GitHub team up to take Git virtual file sys... by jsolson on Friday November 17, 2017 @05:26PM (#15726506)
Piper: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

Unfortunately, it's not available externally.

Even working at Google, my jaw still dropped at this:

    Google's codebase is shared by more than 25,000 Google software
    developers from dozens of offices in countries around the world.
    On a typical workday, they commit 16,000 changes to the codebase,
    and another 24,000 changes are committed by automated systems.
    Each day the repository serves billions of file read requests,
    with approximately 800,000 queries per second during peak traffic
    and an average of approximately 500,000 queries per second each
    workday. Most of this traffic originates from Google's
    distributed build-and-test systems.
Mostly I felt terribly unproductive — my changelist generation rate is way below average.
Re:Microsoft and GitHub team up to take Git virtual file sys... by majewsky on Saturday November 18, 2017 @03:45PM (#15730573)
Can you please quote with ">" at the start of the paragraph instead of using a code snippet? This is unreadable on mobile; the text is 3x as wide as the (scrolling) viewport.
Re:Microsoft and GitHub team up to take Git virtual file sys... by jonhohle on Friday November 17, 2017 @10:20PM (#15727600)
The number of engineers and commits is probably in the same ballpark at Amazon. With a DVCS, far less operations are done server side, so I would imagine the read requests are an order of magnitude or two lower. They have tooling to get a global view across all repos, however there are no cross-repo atomic operations (those can be managed in their build system in a relatively robust way).

What advantage does a company wide mono repo provide?

Re:Microsoft and GitHub team up to take Git virtual file sys... by jsolson on Friday November 17, 2017 @10:48PM (#15727662)
The ability to have something like CitC (or this post's git virtual filesystem) is certainly one big advantage -- no need to clone new packages, they're right there in your "local" source tree. Bazel (blaze) is another, particularly when coupled with working at HEAD.

My experience with farms of git repos is that the lack of atomic operations over many tiny repos leads to things like version sets and having to periodically merge dependencies. I've worked on teams where that was inevitably neglected during hectic periods resulting in painful merges of large numbers of changes. That problem simply doesn't exist with working at HEAD and high quality presubmit test automation/admission control. The single repo also allows for single code reviews spanning multiple packages which makes it MUCH simpler to re-arrange code (Bazel again helps here since a "package" is any directory with a BUILD file). Package creation is lighter weight for the same reason, and has fewer consequences for poor name choices since rearrangement is easy and well supported by automated tools.

Sharing one build system where a build command implicitly spans many packages also results in efficient caching of build artifacts and massively distributed builds (think a distributable and cacheable build action per executable command rather than a brazil-build per package). Each unit test result can be cached and only dependent tests re-run as you tweak an in-progress change. This is fantastic for a local workflow (flaky tests can be tackled with --runs_per_test=1000, which with a distributed build system is often only marginally slower than a single test run). Also, you can query all affected tests for a given change with a single "local" bazel query command. The list goes on from here -- I keep thinking of new things to add (finer grained dependencies, finer grained visibility controls, etc.).

It's not that you can't build most of this for distributed repos, but I'd argue it's harder and some things (like ease of code reorg) are nearly impossible.

Subjectively, having worked with both approaches at scale, Google's seems to result in much better code and repo hygiene.

Re:Microsoft and GitHub team up to take Git virtual file sys... by ryanisnan on Friday November 17, 2017 @07:56PM (#15727175)
Wow! Thanks for sharing. I hadn't contemplated how Google managed their source code. Sounds like Piper has some nice features!
Re:Microsoft and GitHub team up to take Git virtual file sys... by guiambros on Friday November 17, 2017 @08:17PM (#15727261)
Lots more details here [1] if you're interested.

[1] https://www.youtube.com/watch?v=W71BTkUbdqE

Re:Microsoft and GitHub team up to take Git virtual file sys... by sabujp on Friday November 17, 2017 @07:53PM (#15727164)
which means nothing, e.g. by loc your changestats are order of magnitude higher than mine
Re:Microsoft and GitHub team up to take Git virtual file sys... by oconnor663 on Friday November 17, 2017 @05:12PM (#15726423)
Google has one internally, and Facebook has done a ton of work on Mercurial to build something similar, doing things like replacing files on disk with memcache servers running close to the dev machines. But I don't think there's anything that works "out of the box" at that scale.
Re:Microsoft and GitHub team up to take Git virtual file sys... by QuercusMax on Saturday November 18, 2017 @12:44AM (#15727914)
For quite a while at Google there have been git extensions/tools for working with Piper. They're pretty widely used, but haven't been "officially" supported.

There's a new project that allows you to use hg with Piper/CitC; I've been using it for about a month and it's been working really well. No clue how it works, though.

Re:Microsoft and GitHub team up to take Git virtual file sys... by _s on Friday November 17, 2017 @04:49PM (#15726278)
Would you be willing to elaborate on the pros/cons of mono repos vs what git encourages? It seems to just be the wrong fit for a given use case, but works very well others.
Re:Microsoft and GitHub team up to take Git virtual file sys... by ed_blackburn on Friday November 17, 2017 @05:06PM (#15726396)
That's a good and valid point, which deserves a conversation undoubtedly - however, there are a lot of businesses out there who have a large monolith, legacy application gigabytes in size where the cost of slicing and dicing into smaller repositories has to be driven by requirements and not technology.

My current client has a 17-year-old multi Gb TFS monolith. New components go into git and as features evolve it's common for the respective team to hive them off into they're own git repos as stand-alone components. Nevertheless TFS is painful and if I could have an effective git repo with the monolith too, then life would be far easier.

Re:Microsoft and GitHub team up to take Git virtual file sys... by ryanisnan on Friday November 17, 2017 @07:58PM (#15727186)
Is there a way to split a TFS repository, surely you no longer need the past 10 years of commit history on a daily basis - archive it or something?

Also how feasible is migrating from TFS to Git?

Re:Microsoft and GitHub team up to take Git virtual file sys... by Merad on Saturday November 18, 2017 @12:19AM (#15727871)
Very, thanks to git-tfs [1]. My company is in the midst of a move from on premise TFS/TFVC to cloud based VSTS/git. We've run into some projects that managed to somehow mangle their branch history so to the point that we gave up and only migrated the current checkin to git, but most projects pull out the full history and all branches with no trouble.

The only real complaint I have is speed. Dunno if it's our TFS server or the nature of TFS, but every checkin pulled into git takes anywhere from 5-30 seconds. In large projects with long histories you can easily spend 30+ minutes exporting history before running into a problem that prevents the export from completing.

[1]: https://github.com/git-tfs/git-tfs

Re:Microsoft and GitHub team up to take Git virtual file sys... by josteink on Saturday November 18, 2017 @02:22AM (#15728135)
> In large projects with long histories you can easily spend 30+ minutes exporting history before running into a problem that prevents the export from completing.

If it makes you feel any better I had to spend two continuous weeks exporting our huge TFS monolith before I managed to have a working git copy.

But yeah, Git-TFS is a godsend.

Re:Microsoft and GitHub team up to take Git virtual file sys... by hasenj on Friday November 17, 2017 @10:22PM (#15727606)
What I find more surprising is the length they went through to make git work for them.

It would probably have been better if they had assigned a team of highly skilled engineers to design a distributed source control from scratch specifically to solve the problems faced by the windows team.

After all, `git` itself was developed for Linux because Linus was not satisfied with any of the existing solutions.

Re:Microsoft and GitHub team up to take Git virtual file sys... by sverhagen on Friday November 17, 2017 @11:39PM (#15727780)
Why would that have been better?

(Why in the world would that have been better?)

Re:Microsoft and GitHub team up to take Git virtual file sys... by hasenj on Friday November 17, 2017 @11:48PM (#15727805)
For the same reason that git turned out better than svn for a lot of use cases.

It seems like Windows is not really one of these use cases.