News for hackers. Stuff that matters.
Re:Keeping master green at scale by chairleader on Thursday April 18, 2019 @01:45PM (#19693080)
Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."
Re:Keeping master green at scale by huac on Thursday April 18, 2019 @02:25PM (#19693487)
facebook, google, airbnb, quora, many more all use monorepo

obviously there are many others who do not use monorepo (amazon comes to mind) but it's reasonable to claim that they are actually widely used and fundamental when used

Re:Keeping master green at scale by jhenkens on Thursday April 18, 2019 @04:10PM (#19694351)
Microsoft uses it for Windows as well, which was so large they wrote their own git filesystem to power it.
Re:Keeping master green at scale by vruiz on Thursday April 18, 2019 @04:12PM (#19694366)
Does anybody know how these companies development environments look like? I know about Piper at Google but how do the rest manage? Does every single engineer have the entire monorepo in their machines?
Re:Keeping master green at scale by Shish2k on Friday April 19, 2019 @09:10AM (#19699274)
At facebook, a virtual filesystem (https://github.com/facebookexperimental/eden) + change monitor (https://facebook.github.io/watchman/) = source control operations run in O(files I have modified) instead of O(size of repository) time
Re:Keeping master green at scale by vruiz on Friday April 19, 2019 @10:10AM (#19699741)
Very interesting, thanks!
Re:Keeping master green at scale by YawningAngel on Thursday April 18, 2019 @04:29PM (#19694499)
Most places I know of use Git Virtual File System or equivalents.
Re:Keeping master green at scale by vruiz on Thursday April 18, 2019 @04:32PM (#19694530)
It is my understanding that VFSForGit only works on Windows.
Re:Keeping master green at scale by filoleg on Thursday April 18, 2019 @06:35PM (#19695394)
The github repo has instructions for running it on Mac and says that the stable Mac version is under active development.
Re:Keeping master green at scale by venantius on Thursday April 18, 2019 @04:43PM (#19694609)
Airbnb uses a monorepo for JVM-based project but most of Airbnb's code at least as of mid-2017 was not run on the JVM and was hosted multi-repo.
Re:Keeping master green at scale by fjp on Thursday April 18, 2019 @04:57PM (#19694700)
What are AirBnb's unique scaling issues other than just being a web app with tons of usage?

Uber has navigation, route optimization, queuing, etc. Facebook has to propagate activity our to massive and complex social network graphs.

I'm not discounting the toughness of operating at Airbnb's scale, but from my limited understanding it seems like they are not solving a new problem.

Re:Keeping master green at scale by novok on Thursday April 18, 2019 @05:24PM (#19694924)
Map search, automatic dynamic pricing, predicting busy periods, predicting user & provider preferences, etc.
Re:Keeping master green at scale by huac on Thursday April 18, 2019 @05:25PM (#19694931)
i was a data scientist so i knew of monorail but never touched that side of things :)
Re:Keeping master green at scale by twic on Friday April 19, 2019 @06:23AM (#19698327)
Notably, Netflix doesn't (or least didn't):

https://medium.com/netflix-techblog/towards-true-continuous-...

Re:Keeping master green at scale by chairleader on Thursday April 18, 2019 @10:13PM (#19696526)
til! Most of my experience with large, single-repository projects are just plain monoliths. The design goal we strive for tends to be the microservice architecture, assuming that isolation of responsibility leads to more maintainability, better decision making, etc. I can see how, with a well disciplined team, the monorepo could have the best of both worlds.
Re:Keeping master green at scale by bluGill on Thursday April 18, 2019 @03:58PM (#19694257)
Many companies do use a monorepo. Many other companies do not. There are trade offs.
Re:Keeping master green at scale by msangi on Thursday April 18, 2019 @03:24PM (#19694014)
All the companies have in common a huge budget they can invest on their build systems to overcome the shortcomings of monorepos.

They do have some benefits, but they also come with an immense cost

Re:Keeping master green at scale by patejam on Thursday April 18, 2019 @03:38PM (#19694124)
But that's what the OP's quote is saying. "large" companies use them.
Re:Keeping master green at scale by huac on Thursday April 18, 2019 @03:58PM (#19694258)
all of those companies were once small too
Re:Keeping master green at scale by googlemike on Thursday April 18, 2019 @02:18PM (#19693403)
I like it at Google!
Re:Keeping master green at scale by Aqua_Geek on Thursday April 18, 2019 @03:24PM (#19694011)
Google has the team + tooling to properly support it. The same cannot be said for many other orgs.
Re:Keeping master green at scale by glandium on Thursday April 18, 2019 @09:48PM (#19696413)
Google has more people working on the problem than many other companies have employees.
Re:Keeping master green at scale by Aqua_Geek on Thursday April 18, 2019 @10:45PM (#19696684)
I don’t doubt it. They also do more traffic through their VCS than most companies do through their main product.
Re:Keeping master green at scale by erik_seaberg on Thursday April 18, 2019 @10:53PM (#19696720)
Do they? They didn't used to. In 2015 we were routinely dead in the water, unable to test and deploy anything from our google3 projects because some random submitted a CL for a project we didn't even care about. Teams would appoint "build cops" whose job is to complain as quickly as possible because that's all we could do about it.

Every problem you could have with bad dependencies is entirely self-inflicted. The Right Thing™ is to choose a known-good version, and update when you have the bandwidth to pay down the tech debt.

Re:Keeping master green at scale by nevir on Thursday April 18, 2019 @06:56PM (#19695514)
Many teams
Re:Keeping master green at scale by k__ on Thursday April 18, 2019 @04:27PM (#19694490)
Yes, my first thought "I wish the systems I worked on in big corps were big monolithic giants"
Re:Keeping master green at scale by robocat on Thursday April 18, 2019 @05:07PM (#19694783)
Which huge successful software companies don't use a monorepo?
Re:Keeping master green at scale by xxpor on Thursday April 18, 2019 @05:10PM (#19694808)
Amazon.
Re:Keeping master green at scale by slsteele on Thursday April 18, 2019 @11:09PM (#19696813)
I think these count as huge (although maybe not when next to Google), but Spotify and Netflix.
Re:Keeping master green at scale by draw_down on Thursday April 18, 2019 @02:17PM (#19693394)
I'm willing to agree with that premise.