The problem we solve by religiously sticking to "best practice" is the extreme engineering churn in "Silicon Valley" companies.

Not spending 9 months on-boarding might be worth extra minutes on deployment, IDK...

On the other hand, after 9 months on-boarding an engineer is likely to operate much more productively than the "best practice" would allow.

Even more importantly, the engineer is likely to stick with a truly high tech workplace with much less "churn" than in yet another dime a dozen startup with the same fashionable tech stack as the slightly better competition.

Or dig deeper and deeper into a rabbit hole, and come out almost unemployable 7 years later while the entire industry has moved on.

You don't have to have crazy on boarding. In fact, with systems like Kubernetes it can take me months to finally grasp the whole system... especially if you have your own Kubernetes Operator and custom YAML configuration that maps to Kube's... which I have seen.

You can use Fargate for example, which should give you easy to manage and fast deployment times - you just have to figure out the fast build time.

For FastComments.com I can do a build in under two minutes. The second you merge to master:

1. E2E testing runs, which is massively parallelized. Each test suite's data gets namespaced so they don't step on each other.

2. The build runs, takes 30 seconds, because it's not a compiled language. Although if it was in Java I don't think it would take very long.

3. Deployment happens, Orcha sends the bundle to all the servers, stops the deamon, extracts it, restarts the systemd daemon.

4. E2E testing runs again against production as a smoke test.

This all happens in under a couple minutes. Not a very small app - package.json has 50 dependencies and the E2E test suite has ~40 tests. These tests are complicated - some will open two browsers for example to test the live commenting features...

Deploying faster means you can onboard faster, because part of onboarding is deploying.

Deploying faster means you can onboard faster, because you get quicker feedback on your changes, getting insight into code and systems faster.

Deploying faster means you can onboard faster, because you have fewer (ideally just one) pending changes to deploy to keep in your mind while working, increasing the amount of changes you can make in a work week.

Deploying faster means you can onboard faster, because you spend less time deploying and more time studying the system.

I work with a project that takes an hour to build because it’s thousands of C++ files. What would you suggest I do to improve my build times?

Surprisingly, "concatenative builds" where you chuck all the files in a single file and compile that, tend to be a lot faster.

distcc/ccache also works. Years ago we had a rack full of machines devoted to this. Requires a little tuning but lets you do -j 50. Of course, that was in the days of dual-core machines and these days you can buy a lot of cores. Bottleneck is probably writing intermediate files to disk and flushing them.

Untangling the inter-dependencies that cause all the rebuilding is very hard and often forces design changes you don't want, such as use of the "pimpl" idiom.

Even without having programmed in C/C++ for years, it doesn't sound very surprising that "concatenative builds" are faster - I assume it has something to do with the preprocessor having exponentially less work the fewer files you have?

Our QT project had about 400 compilation units (header+source, compiled to a separate object file). Many of them were 100-200 lines of code only. Full build used to take 40 minutes. Each compilation unit pulls in about 40k lines of code from Qt through headers. So the compiler has to process at least 400*40k = 16M lines of code. Concatenative build only processes app code (about 50k) plus about 60k of cumulated framework code. This gives about 100 times speedup. Our concatenative build took indeed less than 1 minute.

This is with unified builds :(

Do you need to recompile all multi-thousand files every time you make a change? Modularising the build would be my first step.

A good build system and caching. By "a good build system" I mean a hermetic one like Bazel that can be confident of knowing the inputs to each build step so that they can be run or not as appropriate. Sometimes you'll touch a core header file and a lot of things will rebuild, but it should still be a lot better.

A buddy slashed his CI build times by using a RAM disk. But yeah, usually faster to avoid compiling things.

We have a similar situation and use ccache and distcc is setup on each dev's workstation. It also helps to use forward declarations instead of including header files whenever possible.

> and one is running tests and packaging software.

and linting, and code quality checks, and compiling, and optimizing, and pushing the package up to package & image repositories, and deploying it to sandboxes

I agree there is room for improvement. Parallelism in particular is a step that could shave up to 30% off some of the build pipelines I've worked with. I disagree however that your example should "make your blood boil". I feel like "Blindly following "best practices" instead of using our $170k+ Silicon Valley brains" is inciteful language that ignores the reality of the situation. We are all working from the tools we've got. The majority of those tools don't have good models for Parallelism. CircleCI for example supports "test splitting" parallelism, where tests can be run in parallel in different containers, but there's no workflow parallelism that I know of so that style checking, deployment to ephemeral environment, publishing, &c can proceed once the bud artifact is generated.

Yes we should try to adapt & grow. The very name, build pipelines, bespeaks a linear process. The ability to change is definitely at hand, & there are legions of us who would pick up the torch on this one & move forward.

But I disagree that our failure to do so is "blindly following 'best practices'". Yes this is one of the "problems you can solve" but it's a bit of a big deal. Configuring build pipelines is a fairly advanced task, & to advance to parallelism & our maybe 30% gains we need not only the machine work to support parallelism, but also the language work to redefine more flexibly how work ought to flow through a system, to build new models for computation that users can author that are graph based, that let work spin up when it's dependencies are defined & satisfied.

"Will most companies let their developers do things that may not be 'best practices' to do that?" Let's start with smaller steps, eh? Should companies let their employees venture forth & tackle new non-linear practices for software assembly? Yes. How many? Every company? No probably not. What responsibilities do you want to put on the team beyond delivery of something that works for them? What documentation do you need? Who reviews the proposed architecture to approve this new best practice you are trying to create? Is this an internal system, something you are going to make a core competency and sell, or are you going to try to really make it a best practice & open source it and try to grow it openly? What best practice tools will you use to develop this async build system, or should developers eskew convention & use their "170k+ Silicon Valley brains" on this (& other) layers too?

There are of course some build/integration/deployment technologies out there that do support wider realms of parallelism. Jenkins with it's groovy syntax at least supports creating stages that can be run in parallel. This is still leaving possible optimizations behind, as all items in a stage have to finish before the next stage can start; it's not fully async. Compromises were made but it works pretty well, helps. I've seen 25% cycle time reductions from parallelism here, it helps.

Overall, to me, this issue feels not so pressing. Engineers scrap together an enormous amount of their own time & effort to do what they can with what is available. They are extremely resourceful. I do indeed want companies that better let their employees pursue process & systems improvements, that spend more time stepping outside of product development & more time improving the house they live under together. Most of all, I hope that we are learning & sharing, openly, publicly, #LearningInPublic, so that the state of the art can improve, so that there is more & better material out there to decide what a best practice might be & how to head there. And how to have it make sense 2 years down the road when all the original engineers have found other opportunities & the new less seasoned crew has to suss out how to proceed.

I also want to touch briefly on the original article, although I'm not sure how. Because it has such a neat exploration of so many interesting rich capabilities, capabilities and that let software manage software: software defined software, software defined systems. And in ways that are outside our newer conventions, where we manage polyglot systems via containerized processes (versus using powerful multi-system vm's to manage). The authors love of empowering the operator with online tools, to let them situate & operate from inside the running machine systems. This is such a glorious & important capability, about how we commune with machines, & author is so right to call attention to what we've lost, to how much less operators are online with their systems & how much more distance there is. Being able to use with the machines is important.

And yet, here too I think the situation has nuance & reason. That it's not only a matter of best practices socially gumming us up & social group think making us dumb. Or, if they are gumming us up, it is in response to another form of getting gummed up. Systems used to be dealt with as pets, as carefully groomed & growing buckets of bits that had acquired various adh-hoc operations & scripts & subsystems over time, as operators went about working on the system & working on systems to help them better work on the system- the author leads with going in depth on the ad-hoc deployment systems that had grown up over the years internally on their system, in just such a manner. When anything is possible & at hand, many things happen. Over time, it's hard for newcomers (& seasoned veterans) to understand what has been done, why things were done, hard to even see what pieces are there. A system that is never killed, that's always carried forward, mutated, ongoingly & organically becomes an unintelligible bucket of bits, an opaque intertwined illegible historical artifact. In contrast, immutable containers don't attempt to strike much of a balance to this situation, don't leave a lot of affordances for many of the wonderful erlang capabilities, but they insure that there is a well know & iterateably improveable way to get from code to deployed code that everyone can observe & understand, that brings predictability & understandability that often, in the past, had been sorely absent.

Please have sympathy. Things are complex. Figuring out not only how to move forwards but move forwards in a robust enduring well-thought-out & mindful manner is a big challenge, & retaining that hunger & appetite for moving forwards, for doing good things is equally important, & navigating ourselves while under this tension is something we have to learn to live with, hopefully peacefully.