How repos are organized is a choice. Google could just as easily (financially speaking) have chosen to break up that repo and use more standard tooling. They did not choose that. Did they have (possibly perfectly valid) reasons for that? Sure.
That does not render it as their only possible path.
Rather than switch their source control strategy to something that would work well with existing products, they chose to build a brand new product to support their existing strategy. This is, IMO, not an obviously sane decision from an outside perspective. Why?
Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
That's how it looks from an outside perspective.
The question I'm asking is not "why didn't they use git?!". The question I'm asking is, "what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?"
Haven't seen an answer to that yet. People are too busy implying or outright stating that I'm stupid for asking the question in the first place.
Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.
Perhaps Google tried to do it this way and hit the same issues.
"The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1]
not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.
A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration?
The answer to that is not obvious.
(edit: response specifically regarding "astronomical" costs)
I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument.
TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate.
I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty.
Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture.
Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice.
I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center.
Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time?
Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence.
If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards?
Yeah, I'm going to call bullshit.
Maybe that is the right decision. But Google has organically grown over the decades and their tooling evolved with that.
It's hard to say which requirements they were working with and what they were trying to optimize for - sometimes higher costs are just okay if they allow or enable other things.
For me it's always "what are we trading?". In an organization the size of Google, it's far easier to amortize away the cost of such a thing. That doesn't mean it's not a cost, though. And there's a lot of tooling around modern SCM that they won't get because they're on a bespoke system, so there's a cost there too.
As you say, at the time the variables may have been very different.
It's just an area where if I had to make that decision (even back then), someone would have had to be exceptionally persuasive to convince me to authorize what is effectively a multi-million dollar project... to recreate source control. =)
> Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool.
Git sucks so much compared to Piper that it boggles the mind to even ask such a question.
I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know.
The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency.
I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue.
It's always a trade; those dollars can potentially do more good somewhere else.
You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH.
It's really hard to deny that Google has acute NIH.
Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions.
I don't think there's anything wrong with that.
My perspective is simple: inventing its own source control system at Google is absolutely worthwhile and the ROI is highly positive. The tradeoff is worthwhile. You might say that before Piper existed it might be a gamble whether it would be worth it. But in hindsight, absolutely 100% yes it's worthwhile. It does not need to be argued for anyone who has used it.
The only thing that is really subject to debate is whether Google's engineering leads had the foresight of what Piper would accomplish or whether they simply wanted to build something custom and it only fortuitously turned out to be good. That's a question no one here can answer.
Bah. Text boxes in browsers suck, and mine's not cooperating at the moment, so instead of writing another book, I'll just say: thank you for the reasonable response. =)
this is the perfect HN comment - so confident but with so little thought.
essentially all google code - something like a billion LOC - are in one repository.
there's something like 100 000 people who use source control, plus jillions of systems doing automatic commits all the time.
it's atomic on each commit.
so that's several QPS globally synchronising a huge data store, with the entire dev tooling system hanging off it.
git is ... very very much not suitable for such systems.
gitlab is just a simple web app reading from a bunch of small and rarely changing git repositories.
Piper is also the successor to Perforce, which is a completely different model to git.