News for hackers. Stuff that matters.
Re:Deno 1.45: Workspace and Monorepo Support by ethegwo on Friday July 12, 2024 @08:03AM (#40944750)
In my last job, I tried using a monorepo approach, merging about 50 microservices into a single repository. After several months of growth, the size of the entire git repository reached over 100GB. We had to create monthly checkpoints (.tar.gz) to facilitate faster time traveling or quick cloning for new team members. This experience taught me that perhaps git is not the best tool for managing a monorepo.
Re:Deno 1.45: Workspace and Monorepo Support by mbb70 on Friday July 12, 2024 @08:17AM (#40944833)
What were you checking in? I've worked in monorepos with a lot of generated code (millions of lines) and 10 years of commits and the total repo size never went over a few gb.
Re:Deno 1.45: Workspace and Monorepo Support by ethegwo on Friday July 12, 2024 @08:57AM (#40945118)
Just golang codes, there are about 500 hundreds developers work on it.
Re:Deno 1.45: Workspace and Monorepo Support by pas on Friday July 12, 2024 @09:02AM (#40945160)
someone else mentioned that the the kernel is still less than 2G
Re:Deno 1.45: Workspace and Monorepo Support by Risord on Friday July 12, 2024 @09:50AM (#40945564)
TLDR: Monorepos have no issues in "real world".

(= My bubble, I am quite sure I haven't ever even met person face to face who have been in project of even 100 devs.)

Re:Deno 1.45: Workspace and Monorepo Support by Capricorn2481 on Friday July 12, 2024 @04:49PM (#40949294)
That's billions of lines of code if it's just Golang. I would imagine most of that was autogenerated code that no one was reviewing. This is not a problem with monorepos.
Re:Deno 1.45: Workspace and Monorepo Support by mananaysiempre on Friday July 12, 2024 @08:13AM (#40944810)
> We had to create monthly checkpoints (.tar.gz)

Not `git bundle`?

Re:Deno 1.45: Workspace and Monorepo Support by danpalmer on Friday July 12, 2024 @08:34AM (#40944951)
There's some truth to this. I'd say that perhaps Git is unsuitable above certain thresholds on numbers of files, numbers of commits, and size of data, but, that the size of data (and the fact that for certain file types this gets blown out on edits) is probably the one most people hit first – it sounds like you may have.

I've heard of git monorepos with >1k microservices, and it sounded like it wasn't a huge problem, but I think it likely had significantly less than 100GB. In fact I feel like you need to be committing large binary files to get towards that size.

Re:Deno 1.45: Workspace and Monorepo Support by azangru on Friday July 12, 2024 @08:50AM (#40945055)
> I'd say that perhaps Git is unsuitable above certain thresholds on numbers of files

Scott Chacon in a recent talk about git[0] said that Microsoft uses git to version control Windows. They have ~3.5 million files in the repo, which is about 300 gigabytes in size. There are some tweaks to improve git performance in those scenarios; but still, I'd say it's a pretty impressive threshold.

[0] - https://www.youtube.com/watch?v=aolI_Rz0ZqY

Re:Deno 1.45: Workspace and Monorepo Support by augusto-moura on Friday July 12, 2024 @08:53AM (#40945090)
There's a lot of "tweaks", they implemented a whole toolchain for virtualizing large repos: https://github.com/microsoft/scalar
Re:Deno 1.45: Workspace and Monorepo Support by re-thc on Friday July 12, 2024 @10:00AM (#40945642)
Most of those "tweaks" are no longer required. A lot of it has been merged back to git itself.
Re:Deno 1.45: Workspace and Monorepo Support by the_duke on Friday July 12, 2024 @08:40AM (#40944993)
How did you manage to grow 50 microservices to 100GB?

A fresh checkout of the whole Linux kernel currently is 1.8GB ...

Were you checking in lots of vendored dependencies? Or 10s of GBs of images/videos/AI models?

Re:Deno 1.45: Workspace and Monorepo Support by hu3 on Friday July 12, 2024 @08:48AM (#40945047)
I'm betting autogenerated file like protobuf definitions.
Re:Deno 1.45: Workspace and Monorepo Support by ethegwo on Friday July 12, 2024 @09:18AM (#40945297)
Yes, there are lots of auto generated thrift templates, we have tried our best to keep deterministic generation.
Re:Deno 1.45: Workspace and Monorepo Support by acedTrex on Friday July 12, 2024 @09:21AM (#40945333)
With that much generation that im assuming changes regularly you almost want a seperate solution or to gitignore them and generate the files at build time...
Re:Deno 1.45: Workspace and Monorepo Support by augusto-moura on Friday July 12, 2024 @08:52AM (#40945081)
How would that differ from having the 50 microservices cloned separately? Maybe you want to say that perhaps some of the microservices were not that useful and didn't need to be versioned as well? If that is the case I think you could benefit from Scalar [1]. Microsoft created it because of similar problems with size and binaries

[1]: https://github.com/microsoft/scalar

Re:Deno 1.45: Workspace and Monorepo Support by slightwinder on Friday July 12, 2024 @09:18AM (#40945291)
> How would that differ from having the 50 microservices cloned separately?

Usually, not everyone needs every single microservice, so they only checkout the handful they need, instead of the full monster. And usually one will accumulate repos over time, instead of getting them all at once. Similar, you will also only pull updates for the repo necessary for your work today, instead of getting the full update the whole company creates every day.

Re:Deno 1.45: Workspace and Monorepo Support by msoad on Friday July 12, 2024 @01:59PM (#40947893)
Sparse checkout is a new Git feature that can help with this
Re:Deno 1.45: Workspace and Monorepo Support by ethegwo on Friday July 12, 2024 @09:11AM (#40945238)
Yes, Scalar would be helpful. The problem is that not every developer needs all the code in the repository for their daily work. However, Git makes it difficult to do partial checkouts.
Re:Deno 1.45: Workspace and Monorepo Support by worldsayshi on Friday July 12, 2024 @07:50AM (#40944672)
I'm currently working in a pnpm monorepo and it really seems like a potential double edged sword. On the one hand I have longed for better support for local packages in the ecosystem (which pnpm does provide, and I'm happy for it). But some weird edge cases seem to arise. Like serverless functions not being supported[1]. I suppose that's just a matter of feature maturity.

The more design-level hazard of this might be that it's easier to build monorepos where everything is tightly coupled. And you end up with something that is like a single package with extra abstraction layers.

1 - https://github.com/pnpm/pnpm/issues/6259

Re:Deno 1.45: Workspace and Monorepo Support by forty on Friday July 12, 2024 @08:16AM (#40944829)
Isn't the problem with serverless only a problem with windows not supporting zipped symlinks ? (From what I understand from the GH ticket)
Re:Deno 1.45: Workspace and Monorepo Support by ricardobeat on Friday July 12, 2024 @09:12AM (#40945245)
Isn’t that what `pmpm deploy` solves? It should copy all packages into the target dir.
Re:Deno 1.45: Workspace and Monorepo Support by worldsayshi on Friday July 12, 2024 @10:46AM (#40946086)
It tries to solve it but doesn't seem to get all the way there. Symlinks are still very much present for example. But I'm still a bit confused what exactly is causing the azure function to fail so I can't say more.
Re:Deno 1.45: Workspace and Monorepo Support by bovermyer on Friday July 12, 2024 @07:59AM (#40944722)
I wonder how this works with things like dependency management; for example, Nx lets you build all affected projects when you change one dependency's files in a monorepo. Does this do something similar?
Re:Deno 1.45: Workspace and Monorepo Support by IshKebab on Friday July 12, 2024 @08:00AM (#40944729)
Deno is so nice for infrastructure scripting in projects. I tried to convince my company to use it but apparently one `curl` line to install it was too much to ask so now I spend half my time fighting Python, setting up venvs, working around pip bugs, etc. Yeay.
Re:Deno 1.45: Workspace and Monorepo Support by rockyj on Friday July 12, 2024 @08:13AM (#40944814)
Not to mention the performance / async / memory usage benefits you would have gotten on Node/Deno (not to mention lower cloud bills). I do not know why teams doing this.
Re:Deno 1.45: Workspace and Monorepo Support by tracker1 on Friday July 12, 2024 @11:11AM (#40946307)
Agreed... I use deno a lot for my ~/bin directory scripts, being able to rely on direct references and just run stuff, without the npm install step is just handy in practice. That node_modules doesn't work the same as well. There are things that are somewhat annoying in practice, but shell scripting with Deno has been beyond nice to say the least.