I would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git.

Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.

It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end.

What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.

Remember "build" time likely also includes test runs, packaging for deployment, ... 40 mins is easy to get to, and has nothing to do with "on a single core".

You can still see which stages are taking a long time.

I wonder if people went into the habit of synchronizing git pushes with socializing breaks, with the proverbial excuse of "yeah, compiling ...".

On day, someone forgot to brief the foreign intern about the necessity of breaks, intern fixes the issue, pointy-hair-boss gets wind of the news, old crew gets fired, new intern gets promoted and fixes also the Pinterest spam on google images.

> new intern gets promoted and fixes also the Pinterest spam on google images

A man can dream.

It takes >12h to build Windows on the MS build platform.

On a single core, Chromium surely takes hours to build.

Though I agree that 40min for the repository in question is highly suspect.

> On a single core, Chromium surely takes hours to build.

Earlier this year Bruce Dawson had a post indicating that it took about a CPU-day, though coalescing files (“jumbo builds”) significantly reduces build time (we’re talking down to 5h), however that’s at the expense of incremental building, and it constrains the code as you can get symbol collisions between the coalesced files.

I read that comment as only applying the "single core" part to the Linux kernel, not Chromium.

I work in games, a clean sync of our project takes well over an hour (and if you're at home it takes multiple hours), and compiling takes a large amount of time. We use lots of tricks - "unity" builds with out build system detecting modified files and compiling them standalone as an example. On my last workstation (2x intel xeon gold) it took about 30 minutes to compile

Yep, I’m shocked that it had to be bloated to 40min before they even thought about fixing it. Anyone who has used Jenkins for nontrivial builds must have had the experience of staring at the slowly expanding session log screen? It doesn’t take any “profiling” to realize git clone’s taking forever.

I remember hearing facebook was on the order of hours (6+) to build.

I would bet on "the new recruit found it, 45 minutes in in the onboarding process".

My first task at my current job, after familiarising myself with the codebase was to improve the CI pipeline

I think everyone has done this, sometimes it really does take a second set of eyes

In my experience it only takes someone who is not drowning in a backlog of tasks yet.

That tends to be the beginners during the onboarding weeks.

The beauty and power of the beginner's mind.

It can see things that were there all along, but everyone who has been there has developed a blindness to.

I see this kind of stuff at companies where only one or two developers work on a project, or the team working on it hasn't had much experience on other projects.

An example would be a company I worked for who ran a pretty standard LAMP setup but had never heard of memcached. Simply adding that reduced the database load by like 90%.

The same way it always happens to me: Over time build time creeps up and you don't really look at this code thinking "I implemented shallow clone years ago so there is nothing I can do, it's slower because we have more code."

Until you or some other person looks what the code is doing.

It could also be that it was a new hire. I shallow clone a huge monorepo similar in commits/branches and it takes seconds. My experience would instantly tell me that something is worth looking into.