News for hackers. Stuff that matters.
Re:Scaling Mercurial at Facebook (2014) by tehlike on Saturday September 12, 2020 @03:19PM (#24454777)
As an engineer that worked both at google and Facebook, I vastly prefer googles monorepo on perforce. Combining that with citc was pretty solid way to develop your project.

Hg is a bit of a nightmare in the wfh situation. Really slow, hangs for a long time if you haven't synced in a few days. Yes, im sure there are ways to tweak, but not sure if you can tweak them enough!

Re:Scaling Mercurial at Facebook (2014) by bolinfest on Monday September 21, 2020 @04:19PM (#24547982)
As a fellow Facebook employee, FYI, Hacker News is not a great place to get tech support.

For the benefit of everyone else on this thread, note that Facebook uses https://github.com/facebookexperimental/eden. From the README:

> Despite having originally evolved from Mercurial, EdenSCM is not a distributed source control system.

i.e., this is not "stock" Mercurial. For example, the Eden server (Mononoke) is written in Rust and the complementary virtual filesystem is written in C++. This has very different performance characteristics than real Mercurial.

EdenSCM works with both EdenFS (the custom filesystem) and a traditional filesystem. If you use EdenFS, pulls will be much cheaper because you only fetch what you use. If you use a traditional filesystem, EdenSCM supports the same "sparse checkouts" feature as stock Mercurial (https://firefox-source-docs.mozilla.org/build/buildsystem/sp...), which can also be used to reduce the size of the slice of the monorepo you pull down.

Last I checked, Perforce (and Google's "implementation" of Perforce, Piper) did not provide nearly the same level of support for stacked diffs as Eden. As both Google and Facebook have cultures of pre-commit code review, working with stacked diffs makes it much easier to make progress while waiting for approvals on earlier diffs.

I believe there are relative advantages/disadvantages of Eden vs. Piper+CitC and that both projects aspire to have the best of each in the limit.

Re:Scaling Mercurial at Facebook (2014) by liveoneggs on Saturday September 12, 2020 @03:58PM (#24455097)
I can only imagine an attempt at the initial clone.. the hg client is ridiculously slow, memory-hungry, and full of dumb behavior that seems to assume local connections
Re:Scaling Mercurial at Facebook (2014) by est31 on Saturday September 12, 2020 @04:13PM (#24455233)
hg has a mode to make initial clones faster by offering zipped compressed checkouts for download.
Re:Scaling Mercurial at Facebook (2014) by liveoneggs on Sunday September 13, 2020 @08:10AM (#24460306)
it does range requests to these bundles, attempts to extract the partials while keeping the http connection alive using up GB of memory in the process, and then totally fails pretty regularly when it need to go back to the network. It makes no attempt at recovering those chunks, re-establishing connections, or using any intelligence at all.

It also overwhelms http header size norms on the regular and runs up memory when attempting to use those bundles.

Chunks/bundles are, in fact, strictly worse unless you get your initial clones with a good piece of software like curl.

Re:Scaling Mercurial at Facebook (2014) by millstone on Saturday September 12, 2020 @10:25PM (#24457674)
When did you work at Facebook? Things used to be bad, but improved greatly after Eden was introduced around 2018. Eden is a FUSE filesystem which uses hg as the backing store.

(I also worked at Google but not the google3 monorepo - instead I got emerge, which was misery.)

Re:Scaling Mercurial at Facebook (2014) by tehlike on Sunday September 13, 2020 @12:53AM (#24458475)
Joined about 6months ago, after 7.5 yrs at google. Eden still doesn't support all repos :(

I do Android occasionally, from laptop, given wfh.

Re:Scaling Mercurial at Facebook (2014) by millstone on Sunday September 13, 2020 @01:32AM (#24458629)
Can you share where Eden doesn't work? I no longer work at Fb but at the time I left Eden was solid for www/android, and mostly there on Mac/iOS.
Re:Scaling Mercurial at Facebook (2014) by Lt_Riza_Hawkeye on Saturday September 12, 2020 @06:34PM (#24456465)
Somehow FB has made all the common operations work in <6 seconds with millions of files...
Re:Scaling Mercurial at Facebook (2014) by gravypod on Saturday September 12, 2020 @08:25PM (#24457141)
It's an impressive feat but something I find even more impressive is the scale they need to be prepared for. The performance challenges of maintaining infrastructure for as many developers as Facebook/Google employ is constant and demanding.

At my extremely small startup in the past ~6 months I've made about 1,000 files (@ HEAD, not including deletions). So, maybe, at the absolute worst case each engineer could produce ~150 files/month. Facebook likely has ~1,000 engineers. That's a worst case 1,800,000 files/year + an existing massive code base.

Good luck to the engineers who are forever battling the scaling issues underlying these orgs! It's extremely impressive to even hear about overcoming these huge hurdles.

Re:Scaling Mercurial at Facebook (2014) by tehlike on Saturday September 12, 2020 @09:32PM (#24457444)
Facebook has way more than 1000 engineers :)
Re:Scaling Mercurial at Facebook (2014) by tehlike on Saturday September 12, 2020 @09:33PM (#24457449)
It all breaks down when it does not because watchman decides to come to a halt and basic arc pull takes more than 10mins after a week of no action on the repo :)
Re:Scaling Mercurial at Facebook (2014) by millstone on Saturday September 12, 2020 @10:25PM (#24457679)
Watchman is gone!
Re:Scaling Mercurial at Facebook (2014) by ian-g on Saturday September 12, 2020 @09:03PM (#24457308)
I'd be very interested to hear what Google's done to their perforce server, if anything.

I used to do vendor support for a company that does version control for chip design. Basically Perforce + some metadata and applications that automate away a bunch of the administrative grunt work.

Some of our larger customers had quite a few layers of abstraction to try and speed it up (to varying degrees of success). But I don't know how Google's scale would compare to a company that would routinely have check-ins they claimed to be hundreds of GB. Or another that checked in their giant PDKs to send them to each of their sites globally

Re:Scaling Mercurial at Facebook (2014) by GauntletWizard on Saturday September 12, 2020 @11:32PM (#24458066)
Google has completely replaced it, in stages, with a whitebox implementation that's by now almost a complete protocol rewrite. You can read about Piper here[1]

[1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

Re:Scaling Mercurial at Facebook (2014) by ian-g on Sunday September 13, 2020 @12:06AM (#24458228)
I've heard about that! Was more thinking about what they did while there was still a Perforce server on the backend
Re:Scaling Mercurial at Facebook (2014) by GauntletWizard on Sunday September 13, 2020 @12:09AM (#24458248)
That I can speak (a little) to: They gave it a gigantic RAM-Backed SAN and the best hardware money could buy. Nothing particularly interesting, but there were several perforce servers with 512G of RAM attached to several SANS with terrabytes of RAM storage (In 2012ish when it was replaced). It was only ever explained to me in terms of SWE-Hours, but my understanding is that the hardware budget was enough to fund the several software engineering teams that replaced it, and the licensing was more than the hardware.