News for hackers. Stuff that matters.
Re:The largest Git repo by vtbassmatt on Wednesday May 24, 2017 @12:59PM (#14411502)
Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/

Summary:

- Complicates daily life for every engineer

- Becomes hard to make cross-cutting changes

- Complicates releasing the product

- There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would still have been a similar order of magnitude in most dimensions

Re:The largest Git repo by erikpukinskis on Wednesday May 24, 2017 @02:00PM (#14412071)
> - Becomes hard to make cross-cutting changes

This does seem like a negative, doesn't it?

But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo.

It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be.

It puts pressure on you to really, actually separate concerns. Not just put "concerns" into a source file that can reach into any of a million other source files and twiddle the bits.

"Easy to make sweeping changes" really means "easy to limp along with a bad architecture."

I think that's one of the reasons why so much code rots: developers thinking it should be easy to make arbitrary changes.

No, it should be hard to make arbitrary changes. It should be easy to make changes with very few side effects, and hard to make changes that affect lots of other code. That's how you get modules that get smaller and smaller, and change less and less often, while still executing often. That's the opposite of code rot: code nirvana.

Re:The largest Git repo by Goronmon on Wednesday May 24, 2017 @02:39PM (#14412363)
No, it should be hard to make arbitrary changes.

If you change the word "arbitrary" to "necessary" (implying a different bias than the one you went with) then all of a sudden this attitude sounds less helpful.

Similarly "easy to limp along with a bad architecture" could be re-written as "easy to work with the existing architecture".

At the end of the day, it's about getting work done, not making decisions that are the most "pure".

Re:The largest Git repo by colechristensen on Wednesday May 24, 2017 @04:21PM (#14413133)
You have to balance getting work done vs. purity, and Microsoft has spent years trying to fix a bad balance.

Windows ME/Vista/8 were terrible and widely hated pieces of software because of "getting things done" instead of making good decisions. They made billions of dollars doing it, don't get me wrong, but they've also lost a lot of market share too and have been piling on bad sentiment for years. They've been pivoting and it has nothing to do with "getting work done" but by going back and making better decisions.

Re:The largest Git repo by contextfree on Wednesday May 24, 2017 @07:37PM (#14414292)
Those releases (well, Vista and 8 anyway, I don't know about ME) came out of a long and slow planning process - if they made bad decisions I don't think it was about not taking long enough to make them.
Re:The largest Git repo by ern on Wednesday May 24, 2017 @06:23PM (#14413931)
I assumed that Windows 8 was hated because it broke the Start Menu and tried to force users onto Metro.
Re:The largest Git repo by sjwright on Thursday May 25, 2017 @05:06AM (#14416148)
It also broke a lot of working user interfaces, e.g. wireless connection management.
Re:The largest Git repo by erikpukinskis on Wednesday May 24, 2017 @06:51PM (#14414057)
> At the end of the day, it's about getting work done, not making decisions that are the most "pure".

This attitude will lead to a total breakdown of the development process over the long term. You are privileging Work Done At The End Of The Day over everything else.

You need to consider work done at every relevant time scale.

How much can you get done today?

How much can you get done this month?

How much can you get done in 5 years?

Ignore any of these questions at your peril. I fundamentally agree with you about purity though. I'm not sure what in my piece made you think I think Purity Uber Alles is the right way to go.

Re:The largest Git repo by nimchimpsky on Wednesday May 24, 2017 @07:45PM (#14414318)
> This attitude will lead to a total breakdown of the development process over the long term.

As evidenced by Microsoft following the one repo rule and not being able to release any new software.

Wait, what ?

Re:The largest Git repo by erikpukinskis on Wednesday May 24, 2017 @08:10PM (#14414427)
The text I quoted had nothing to do with monolithic repos.
Re:The largest Git repo by taeric on Wednesday May 24, 2017 @07:25PM (#14414242)
The linux codebase exists in stark contrast to your claim. Assuming your claim is that broken up repos is the better way.
Re:The largest Git repo by hueving on Wednesday May 24, 2017 @07:52PM (#14414341)
No it doesn't. I think you are thinking of the kernel, which is separate from all the distros.

Linux by itself is just a kernel and won't do anything for you without the rest of the bits that make up an operating system.

Re:The largest Git repo by taeric on Wednesday May 24, 2017 @09:41PM (#14414763)
Then I'll point to the wide success of monolithic utilities such as systemd as evidence that consolidating typically helps long term.

Which is to say, not shockingly, it is typically a tradeoff debate where there is no toggle between good and bad. Just a curve that constantly jumps back and forth between good and bad based on many many variables.

Re:The largest Git repo by hueving on Saturday May 27, 2017 @04:32AM (#14429393)
systemd is also completely useless on its own. It still needs a bootloader, a kernel, and user-space programs to run.

When it comes to process managers, there is obviously disagreement about how complex they should be, but systemd is still a system to manage and collect info about processes.

Re:The largest Git repo by caf on Wednesday May 24, 2017 @08:59PM (#14414640)
The hierarchical merging workflow used by the Linux kernel does mean that there's more friction for wide-ranging, across-the-whole-tree changes than changes isolated to one subsystem.
Re:The largest Git repo by taeric on Wednesday May 24, 2017 @09:43PM (#14414771)
Isolated changes will always be easier than cross cutting ones. The question really comes down to whether or not you have successfully removed cross cutting changes. If you have, then more isolation almost certainly helps. If you were wrong, and you have a cross cutting change you want to push, excessive isolation (with repos, build systems, languages, whatever), adds to the work. Which typically increases the odds of failure.
Re:The largest Git repo by vec on Wednesday May 24, 2017 @04:39PM (#14413268)
Arguing about purity is only pointless and sanctimonious if the water isn't contaminated. Being unable to break a several hundred megabyte codebase into modules isn't a "tap water vs bottled" purity argument, it's a "lets not all die of cholera" purity argument.
Re:The largest Git repo by vtbassmatt on Wednesday May 24, 2017 @02:33PM (#14412317)
As the linked article says, modularizing and living in separate repos was the plan of record for a while. But after evaluating the tradeoffs, we decided that Windows needs to optimize for big, rapid refactors at this stage in its development. "Easy to make sweeping changes" also means "easy to clean up architecture and refactor for cleaner boundaries".

The Windows build system is where component boundaries get enforced. Having version control introduce additional arbitrary boundaries makes the problem of good modularity harder to solve.

Re:The largest Git repo by Eyas on Wednesday May 24, 2017 @03:21PM (#14412712)
You say that, but it is very telling that every large company out there (Google and Facebook come to mind) go for the single-repository approach.

I'm sure that, when dealing with stakeholder structures where different organizations can depend on different bits and pieces, having multiple repositories with difficulty of making breaking and cross-cutting changes, becomes good.

From the view of a single organization where the only users of a component are other components in the same organization, it seems like there is consensus around single-repository.

Re:The largest Git repo by Jach on Wednesday May 24, 2017 @04:11PM (#14413059)
It is very telling. Google has a cloud supercomputer doing nothing but building code to support their devs. I don't know about Facebook. (I really don't -- I'm constantly amazed that they have as many engineers that they do, what do they all work on?) Where I work (https://medium.com/salesforce-engineering/monolith-to-micros...) there's a big monolith but with more push towards breaking things up, at the architecture level and also on the code organization level. We also use and commit to open source projects (that use git), so to integrate those with the core requires a bit more effort than if they were there already but it's not a big burden and the benefits of having their tendrils being self-contained are big.

Which brings me to the point that in the open source world, you can't get away with a single-repository approach for your large system. And that also is telling, along with open source's successes. So which approach is better in the long run? I'd bet on the open source methods.

Re:The largest Git repo by flai on Wednesday May 24, 2017 @05:38PM (#14413672)
You forget Conway's law:

> organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations

Open source has a very different communication structure than a company. While the big three (MS, Google, FB) try to work towards good inter-departmemt relations, it is usually either

- a single person - a small group

that are the gatekeepers for a small amout of code, typically encapsulated in a "project". They do commit a lot to their project, yet rarely touch other projects in comparision.

Also, collaboration is infinitely harder, as in the office you can simply walk up to someome, call them, or chat them - in OSS a lot of communication works via Issues and PRs, which are a fundamentally different way to communicate.

This all is reflected by the structure of how a functionality is managed: Each set of gatekeepers gets their own repository, for their function.

Interestingly this even happens with bigger repositories: DefinitelyTyped is a repository for TypeScript "typings" for different JS libraries, which has hundreds of collaborators.

Yet, if you open a pull-request for an existing folder the ones that have previously made big-ish changes can approve / decline the PR, so each folder is its own little repo.

https://github.com/DefinitelyTyped/DefinitelyTyped/pull/1672...

So: maybe the solution is big repos for closed companies, small repos for open-source?

Re:The largest Git repo by aanm1988 on Wednesday May 24, 2017 @04:35PM (#14413231)
Amazon doesn't.
Re:The largest Git repo by likpok on Wednesday May 24, 2017 @02:50PM (#14412449)
From my experience with Java-style hard module dependencies, this makes it extremely difficult to refactor anything touching external interfaces.

You say this forces you to think ahead, but predicting the future is quite difficult. The result is that you limp along with known-broken code because it would take so much effort to make the breaking changes to clean it up.

For example, lets say you discover that people are frequently misuing a blocking function because they don't realize that it blocks.

Let's say that we have a function `bool doThing()`. We discover that the bool return type is underspecified: there's a number of not-exactly-failure not-exactly-success cases. In a monorepo, it's pretty easy to modify this so that `doThing()` can return a `Result` instead. With multiple repos and artifacts, you either bring up the transitive closure of projects, or you leave it for someone to do later. For a widely used function, this can be prohibitive. That makes people frequently choose the "rename and deprecate" model, which means you get an increasing pile of known-bad functions.

Re:The largest Git repo by johnfn on Wednesday May 24, 2017 @04:27PM (#14413172)
Have you actually worked on a repository at Windows scale..? If not, how can you know that your guesses about the workflow are accurate?
Re:The largest Git repo by emmelaich on Wednesday May 24, 2017 @08:59PM (#14414639)
Making something difficult even more difficult is not helpful to anyone.

And what happens if the inherent difficult disappears or reduces? You're still left with the imposed external difficulty.

Re:The largest Git repo by phkahler on Wednesday May 24, 2017 @02:33PM (#14412318)
Remember when they said IE was an integral part of the OS? Yeah...
Re:The largest Git repo by electrum on Wednesday May 24, 2017 @01:53PM (#14412003)
Facebook cited similar reasons for having a single large repository: https://code.facebook.com/posts/218678814984400/scaling-merc...
Re:The largest Git repo by rbanffy on Wednesday May 24, 2017 @04:13PM (#14413072)
I remember that, in the 90's, you'd often get new UI elements in Office releases that then would eventually move into Windows. There was a technical reason - the cross-cutting - but there also seemed to be a marketing reason - the moment those UI elements became part of core Windows, all developers (yours truly included) would be able to use those elements, effectively negating Office the fresh look before the competition.
Re:The largest Git repo by afpx on Wednesday May 24, 2017 @01:54PM (#14412012)
That's a really interesting article. I wish I would have found it before going down a similar path with my team, recently.

These types of use cases seem so commonly encountered that there should be a list of best practices in the Git docs.

Re:The largest Git repo by e40 on Wednesday May 24, 2017 @05:33PM (#14413640)
It's harder to share code between repos, though.

EDIT: like if something was to be shared between Windows and Office, for example.

Re:The largest Git repo by cosinetau on Wednesday May 24, 2017 @01:46PM (#14411928)
This was a very interesting point. It sounds like there are some serious architectural limitations on Windows, and this makes me believe the same might be true for the NT kernel, and that MS might not be interested in doing heavy refactoring of it.

I'm not a frequent Windows user, or a Windows dev at all. Does anyone know of any consequences that MS's decision might mean, if this hypothesis is true?

Re:The largest Git repo by nxc18 on Wednesday May 24, 2017 @02:50PM (#14412454)
The NT kernel is surprisingly small and well-factored to begin with - it is a lot closer to a 'pure' philosophy (e.g. Microkernel) than something like Linux to begin with.

If you have a problem with Windows being overcomplicated or in need of refactor it is almost certainly something to do with not-the-kernel.

If you look at something like the Linux kernel its actually much larger than Windows. It needs to have every device driver known to man (except that one WiFi/GPU/Ethernet/Bluetooth driver you need) because internally the architecture is not cleanly defined and kernel changes also involve fixing all the broken drivers.

Re:The largest Git repo by caf on Wednesday May 24, 2017 @09:09PM (#14414666)
Small and well-factored the core kernel may be, but if you're parsing fonts in kernel mode, you ain't a microkernel.

( https://googleprojectzero.blogspot.com.au/2015/07/one-font-v... )

Re:The largest Git repo by nxc18 on Wednesday May 24, 2017 @11:20PM (#14415091)
For sure, and Windows is not a microkernel, but it does have separated kernel-in-kernel and executive layers; it would approach being a microkernel architecture if the executive was moved into userland. This is similar to how macOS would be a microkernel, if everything wasn't run in kernel mode (mach, on which it is partially based, is a microkernel).

Of course the issue here is that after NT 4, GDI has been in kernel mode; this is necessary for performance reasons. Prior to that it was a part of the user mode Windows subsystem.

I'd be curious to see if GDI moved back to userland would be acceptable with modern hardware, but I suspect MS is not interested in that level of churn for minimal gain.

Re:The largest Git repo by andrey_utkin on Wednesday May 24, 2017 @05:25PM (#14413583)
Could you please share a link to NT kernel sources so that I take a look?

It is not true that internal architecture of Linux drivers is not clearly defined. It is just a practical approach to maintenance of drivers (as an author of one Linux hardware driver I'm pretty sure the best possible). Reasoning is outlined in famous "Stable API nonsense" document http://elixir.free-electrons.com/linux/latest/source/Documen...

I don't think Windows approach is worth praising here. It results in a lot of drivers stopping working after few major Windows release upgrades. In Linux, available drivers can be maintained forever.

Re:The largest Git repo by nxc18 on Wednesday May 24, 2017 @06:25PM (#14413941)
If you are sufficiently motivated, NT 4 leaked many years ago and you could find it; it even has interesting things like DEC Alpha support & various subsystems still included IIRC. Perhaps you could find a newer version like 5.2 on GitHub or another site, but beware, as a Linux dev/contributor, you probably don't want to have access to that.

FWIW, I've stumbled upon both of those things in my personal research while I had legitimate access to the 5.2 sources as a student. It turns out Bing will link you directly to the Windows source code if you search for <arcane MASM directive here>.

Yes, I'm in the process of reporting this to Microsoft and cleansing myself of that poison apple.

Re:The largest Git repo by GlobalServices on Wednesday May 24, 2017 @03:24PM (#14412733)
Yes! Windows Kernel is a much more "modern" microkernel architecture than any of the circa-1969 Unix-like architectures popular today. We use Windows 10 / Windows Serve for everything at our company, and we have millions of simultaneously connected users on single boxes. No problems and easy to manage.
Re:The largest Git repo by Analemma_ on Wednesday May 24, 2017 @01:52PM (#14411993)
It seems presumptive to say that "If you can't use multiple repos, your architecture must be bad". I could just as easily counter with, "If you're using multiple repos, it must mean you have an unnecessarily complex and fragile microservice architecture"
Re:The largest Git repo by cosinetau on Wednesday May 24, 2017 @02:09PM (#14412129)
I'm sorry that was implied. I simply want better insight into the kernel from people who have experience developing large kernels, and the decisions that are made as a consequence of architectural choices.

I feel like that those questions are valid, and are important in this field, not just kernel development. As someone who desires to continue learning, I will not yield to your counter.

Re:The largest Git repo by planteen on Wednesday May 24, 2017 @01:48PM (#14411950)
Google also uses a single giant repo...
Re:The largest Git repo by monksy on Wednesday May 24, 2017 @02:03PM (#14412087)
Why is that a "good reason" to do it?
Re:The largest Git repo by sateesh on Wednesday May 24, 2017 @02:15PM (#14412176)
Not really :) and I am not the OP. This article [1] provides a very good overview of the repository organization Google has and the reasons behind it.

I think the reason that this works for well Google is the amount of test automation that is in place which seems to be very good at flagging any errors due to dependency changes before it gets deployed. Not sure how many organizations have invested and have an automated test infrastructure like Google has built.

1. https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

Re:The largest Git repo by cromwellian on Wednesday May 24, 2017 @06:59PM (#14414099)
It radically simplifies everything. Every commit is reproducible across the entire tool chain and ecosystem.

It makes the entire system kind of pure-functional and stateless/predictable. Everything from computing which tests you need to run, to who to blame when something breaks, to caching build artifacts, or even sharing workspaces with co-workers.

While this could be implemented with multiple repros underneath, it would add much complexity.

Re:The largest Git repo by yesiamyourdad on Wednesday May 24, 2017 @03:45PM (#14412883)
I think this is like the "your data isn't big enough for HDFS" argument earlier this week. The point I take away is that at some stage of your growth, this will be a logical decision. I don't think it implies that the same model works for your organization.
Re:The largest Git repo by cwyers on Wednesday May 24, 2017 @01:55PM (#14412019)
And Facebook.