We Put Half a Million Files in One Git Repository, Here’s What We Learned

Re:We Put Half a Million Files in One Git Repository, Here’s... by haizzz on Thursday June 16, 2022 @02:33AM (#31763046)

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository.

The content of these translation files are snapshot in time aligned with the text in our product so simply removing them we would lose all the changes made to translations each time texts are changed.

Sorry about that, hope this clears it up!

Edit: for more information on translation and xlf files, we have another blog post all about them https://canvatechblog.com/how-to-design-in-every-language-at...

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by prmoustache on Thursday June 16, 2022 @04:26AM (#31763683)

Can't those xlf files be stored in a separate repo or in an object storage and let the build system fetch them from there?

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by necovek on Thursday June 16, 2022 @06:16AM (#31764320)

That wouldn't be called a "monorepo" then :D

Obviously, the problems they are solving (and admitting to solving) are due to their dedication to the monorepo.

With all the effort spent on working around the drawbacks, I really wonder what advantages they are seeing that make it worth their while?

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by prmoustache on Thursday June 16, 2022 @04:42PM (#31770735)

Well, there is monorepo and monorepo. Git was primarily made to host code, not necessary artifacts. I would categorize those files as artifacts and in my opinion it would still be a code monorepo to have everything else on a single repo.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by necovek on Friday June 17, 2022 @02:34AM (#31774772)

They are code files managed by a different set of people (translators), not generated artifacts (as they explain elsewhere in this discussion).

If we are being pedantic, git was not designed to host multiple projects in a single repo (otherwise, git would have been a subdirectory in the kernel tree). But tools are made without knowing how they'll be used, and that's ok, so I wouldn't stress on what the purpose for monorepo was, but how it's used and what value it brings.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by prmoustache on Friday June 17, 2022 @11:51AM (#31780016)

>They are code files managed by a different set of people (translators), not generated artifacts (as they explain elsewhere in this discussion

Translations do not look like code to me. Rather human generated artifacts.

What I understand is there is a strong tie/match needed between versions of these translations files and the code itself, so I believe this is where having all in the same repo would make sense, having the translators update those file when code has been modified...

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by ghusbands on Friday June 17, 2022 @09:54AM (#31778186)

I think it's relatively normal to include build artifacts in a monorepo when you don't want users to have to be able to build every single one. Especially if you don't want to have to buy a license for every developer for specialist software that only needs to be used by a few people.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by TameAntelope on Thursday June 16, 2022 @01:07PM (#31768451)

Given the religious nature of the "monorepo" vs. "not-monorepo" argument, I would venture a guess that suggesting storing these xlf files elsewhere would upset someone's notion of what a "monorepo" Should Properly Be.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by klysm on Thursday June 16, 2022 @07:08AM (#31764643)

Having them under version control seems very important. Monorepo makes that easy

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by fomine3 on Thursday June 16, 2022 @03:09AM (#31763282)

Thanks for additional info. I'm happy to hear that localization is on your priority.

> English source strings used in our frontend code live in Typescript files (.messages.ts). Source strings used in our backend code live in Java files (Message.java). Our internationalization (i18n) pipeline converts this into a series of XLIFF files (ending in .xlf), with one file for each locale. All these files live in the repository, but the translated .xlf files should never be modified by hand since they are updated automatically when strings get translated.

Isn't it cache?

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by haizzz on Thursday June 16, 2022 @03:17AM (#31763320)

Not quite, the .messages.ts and Message.java files are only in English which are then converted into XLIFF files for other languages so they're the source of truth for what the string should be in different locales so they're not cache

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here’s... by sally1620 on Thursday June 16, 2022 @11:47PM (#31773904)

If it is output of some other tool, then it is a build artifact and there is no need to check it in.

I have this argument with people all the time and the conclusion is always like: "it is too hard to integrate the generator with the build system so we check them in".

The big problem with generated files is merge conflicts. How do you resolve a merge conflict on generated files. especially if they are binary.

[ Parent | on HN ]