It was faster to just sync the workspace over the internet than it was to create the volume from the snapshot, and a clean build was quicker from the just sync’ed workspace than the snapshotted one, presumably to do with however EBS volumes work internally.
We just moved our build machines to the same VPC as the server and our download speeds were no longer an issue.
Its a lot faster in my case (little over 3TiB for latest revision only).
[0] https://help.perforce.com/helix-core/server-apps/p4vfs/curre...
You could have possibly had existing volumes with mostly up to date workspaces. Then you're just paying for the attach time and the sync delta.
My experience with running a c++ build farm in the cloud is that in theory all of this is true but in practice it costs an absolute fortune, and is painfully slow. At the end of the day it doesn’t matter if you’ve provisioned io1 storage; you’re still pulling it across something that vaguely resembles a SAN, and that most of the operations that AWS perform are not as quick as you think they are. It took about 6 minutes to boot a windows ec2 instance, for example. Our incremental build was actually quicker than that, so we spent more time waiting for the instance to start up and attach to our volume cache than we did actually running CI. The cost of the machines was expensive that we couldn’t justify keeping them running all day.
> You could have possibly had existing volumes with mostly up to date workspaces.
This is what we did for incremental builds. The problem was when you want an extra instance that volume needs to be created. We also saw roughly a 5x difference in speed (IIRC, this was 2021 when I set this up) between a noop build on a mounted volume and a noop build that we had just performed the build on.
It was faster to just not do any of this. At my current job we pay $200/mo for a single bare metal server, and our CI is about 50% quicker than it was for 20% of the price.
[0]: https://www.kernel.org/best-way-to-do-linux-clones-for-your-...
[1]: https://web.git.kernel.org/pub/scm/linux/kernel/git/mricon/k...
however, it seems to use the git bundle technique mentioned in the article.
[1] https://www.reddit.com/r/linux/comments/2xqn12/im_part_of_th...
However, the shell script they use doesn't have the bug that I submitted a patch to address - it should have all the refs that were bundled.
I cloned the repo, then was doing occasional `git fetch origin main` to keep main fresh - so far so good. At some point I wanted to `git rebase origin/main` a very outdated branch, and this made git want to fetch all the missing objects, serially one by one, which was taking extremely long compared to `git fetch` on a normal repo.
I did not find a way to to convert the repo back to "normal" full checkout and get all missing objects reasonably fast. The only way I observed happening was git enumerating / checking / fetching missing objects one by one, which in case of 1000s of missing objects takes so long that it becomes impractical.
It seems like large GitHub repos get an ever-growing penalty for GitHub exposing `refs/pull/...` refs then, which is not great.
I will do some further digging and perhaps reach out to GitHub support. That's been very helpful, thanks Scott!
> An immediate benefit of the new protocol is that it enables reference filtering on the server-side, this can reduce the number of bytes required to fulfill operations like git fetch on large repositories.
[1] https://github.blog/changelog/2018-11-08-git-protocol-v2-sup...
microsoft/git is focused on addressing these performance woes and making the monorepo developer experience first-class. The Scalar CLI packages all of these recommendations into a simple set of commands.
https://github.com/microsoft/scalarIt doesn't address the issue of "how to clone entire 10GB with full history faster". (Although it facilitates sparse checkouts, which can be beneficial for "multi-repos" where it makes sense to only clone a part of repo, like in old good svn.)
Semi-related: I always wondered but never got time to dig into what exactly are the contents of the exchange between server and client; I sometimes notice that when creating a new branch off main (still talking the 1M commits repo), with just one new tiny commit, the amount of data the client sends is way bigger than I expected (tens of MBs). I always assumed the client somehow established with the server that it has a certain sha, and only uploads missing commit, but it seems it's not exactly the case when creating a new branch.