We manage daily, but even that is a miracle. If you're building enterprise CRUD app #310979707 or SaaS project #2009879, sure, deploy constantly. But a very significant number of projects simply cannot do that.
I see that I had assumed we were talking about web apps. This assumption narrowed my perspective, and was an error in thinking on my part.
This is great until it's not. We're starting to hit this tipping point on our code-base.
The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features.
Instead, we do occasional freezes for important releases. Not sustainable, but it's a good enough bridge.
The occasional freezes mean future tests written do not need to be optimised (there will be a freeze for important releases anyway), so the unoptimised-tests problem will grow worse the longer you use this strategy, and the cost to rectify it will grow, until your release frequency collapses on itself and everyone complains that "there's no solution to this problem".
These are the sorts of things I think it's very important to nip in the bud. Spend the two weeks now, when it only takes two weeks! Much better than putting everything on hold for six months to fix it four years from now when it becomes untenable.
That does seem like a possibly worthwhile choice though? A "stop chopping the tree to sharpen the axe" moment?
But, maybe spend a day on it. I've been to places where the build took hours and there were obvious 30min solutions to fix it (which I did, in some cases).
Consider it an investment. 2 weeks now gives you how many more features over the next six months due to increased dev velocity?
I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)
caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed.
generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time
The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time.
That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter.
The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility.
I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.
E.g. we went with that route with GH Actions, until our tests didn't fit into the memory of the provided machine sizes anymore (shouldn't be a problem anymore as GH provides options for bigger runners now). So we had to spend 1-2 days on setting up a self-hosted runner in a way that it wouldn't be too heavy in maintenance, and then probably another 1-2 days of rework over the next few weeks to iron out the kinks.
Use feature flags to decouple Deployment from Release and you’re fine.
But management sees the capability and says add a flag for customer Y that changes how this page works.
And after it’s enabled for everybody, remove the flag.
1. Pre Series A companies, which didn't have strong engineering team from start and now struggle to ship. 2. Idea/Pre Seed companies, which just want to ship fast and raise funds. That kind of founder usually have 'friends & fools' budget.
1. First kind of projects usually don't have enough tests coverage to deploy safely more often. Typically I would set a goal to deploy twice a week in 6-12 months and build a plan of how to achieve that. 2. Usually I advice to not write lots of tests for the first ~6 months to focus on shipping faster. In my experience products often change dramatically during this period. But a) there are exceptions. Some domains (e.x. finance) are critical to write more tests. b) tests infrastructure are always set as part of CI/CD process, so engineers can decide and write them for area of high importance.
Bad tests can absolutely make you slower. Good tests can be existential.
At an early stage startup with a small app it might not be as big a problem, but where I'm at right now the app is over 10 years old and our test suite with many many parallel workers is still well over an hour cycle time to get that sanity feedback on a commit, and it's painful.
The benefit is that the tests highlight what changed in the app. So when you update a library and 1+1 changes, it’s an alarm bell to investigate further because you didn’t expect this to happen.
Test_OnePlusOneEqualsTwo()
is not as useful as
Test_CanAddTwoNumbersTogether()
Test for behaviour, not implementation.
This is the important point.
Anyone talking about code coverage percentages is already making a mistake. percentage of coverage isn't nearly as important as WHAT is covered, and it's not a good proxy for it either.
(I'm not trying to imply that that's a mistake you've made; it's just something that it's easy to forget as a manager of programmers, and is maybe worth pointing out explicitly; you need a way to measure the team's productivity, but it's very hard to find a good, measurable proxy for 'productivity' which isn't eventually going to end up being gamed by the whole team!)
I would not judge an individual developer's productivity on commits per day, agree that would be foolish.
I very much agree with break features into many small pull requests. We merge constantly even (especially) if the code path won't be hit yet. Just get it into production. Makes the code review faster, the complexity lower, and lets you focus on one thing at a time.
I think no tests makes you slower. We paid a bit of cost to start writing tests, but now we can deploy 10 times a day because we know the mission critical stuff is working. It's more about testing the right stuff. Decide what can break, and leave it untested. What can't break should be covered by tests. We do stuff with money, all that is robustly tested. The UI isn't.
Code freezing / weekly releasing is counter to CI/CD. Just make your deployment take less than 10 minutes, and ship code constantly.