Re:DevOps: An idea so good, no one admits they don’t do it
by wizofaus on Monday August 29, 2022 @04:51PM
(#32643390)
Yes I do get the general argument for CD (though I have issues with how universally it can be applied based on the sort of BVT I've had to do for SaaS platforms I worked on in the past, where it just wasn't possible to safely release new versions of key components without needing several days' worth of testing, most of which was waiting for verification that interactions with banks etc. had worked as expected).
Re:DevOps: An idea so good, no one admits they don’t do it
by theptip on Tuesday August 30, 2022 @12:56AM
(#32646765)
Agreed, like almost everything it’s not universally applicable.
If you have external contracts that are onerous to validate then it can be quite expensive to build high-fidelity internal representations of those contracts. Could still be ROI positive to do so!
In most cases the contracts are either internal or simple (non-temporal) external contracts like a REST API, so most orgs benefit from moving in this direction.
If you're not familiar with the general arguments for why CD might be desirable, a good place to start would be the books Continuous Delivery [https://smile.amazon.com/Continuous-Delivery-Deployment-Auto...] or Accelerate [https://smile.amazon.com/Accelerate-Software-Performing-Tech...]. The TLDR would be that most deploy problems in practice turn out to be because the process is manual, infrequently-used, and/or unreliable, and by practicing "if it hurts, do it more often" you are forced to iron out those kinks. For example you can't deploy automatically without a good acceptance/smoke test suite, and without solid DB migration testing/linting/practices, to name just two of many. You could just build all these things without trying to do CD, but in practice it seems that it's quite hard for organizations to prioritize this work in isolation, and the goal of CD seems to provide a fairly clear map to a much higher-quality deploy process. The practice of deploying more frequently to flush out issues, is also quite unintuitive as you note.
Here's a (probably out-of-date) article on how FB did deploys in 2017: https://www.infoq.com/news/2017/09/facebook-release-scale/ -- batched commits and staged canary rollouts automatically cut and released every few hours.