Rapid release at massive scale

Re:Rapid release at massive scale by trjordan on Friday September 01, 2017 @02:02AM (#15146478)

One of the undersold (imho) parts of this post is the system that allows tiering of releases with gates for their release. When most companies talk about CI/CD, they mean that master gets deployed to production, full stop. Rollbacks mean changing the code. In reality, when code hits master, there is ALWAYS a lag while it gets deployed, and it's worth having a system that holds that source of truth. Where release engineering gets interesting is how you handle the happy path vs. a breaking release.

I like that Facebook separated out deploy from release. It means that you can roll the release out relatively slowly, checking metrics as you go. Bad metrics mean blocking the release, which means turning off the feature via feature flag. I think for the rest of the world, that would mean halting the release and notifying the developer.

Disclosure: I work with smart people who spend lots of time thinking about this and writing blog posts like "Deploy != Release": https://blog.turbinelabs.io/deploy-not-equal-release-part-on...

[ Parent | on HN ]

Re:Rapid release at massive scale by rdsubhas on Friday September 01, 2017 @07:42AM (#15147565)

This is true, except it has a huge underlying requirement: that all deployments are forwards and backwards compatible. i.e. a running service must be able to talk to the older version of itself, and vice versa (and of course the chain of dependencies). This is a much bigger knowledge investment, easier said than done.

It pays off in the end, but not worth making it a "criteria for success" when breaking out from branch-based to trunk-based continuous delivery, otherwise the trunking will most likely end up never happening.

shameless plug: at goeuro.com we shifted from branch-based to trunk-based CD in a short time (<3 months) with a diverse set of services and workloads, by applying a holistic socio-cultural, technical and process approach. Could be of interest if anyone is trying to make a switch: https://youtu.be/kLTqcM_FTCw

[ Parent | on HN ]

Re:Rapid release at massive scale by bpicolo on Friday September 01, 2017 @08:09AM (#15147667)

> that all deployments are forwards and backwards compatible

This is critical regardless in SoA / anything other than strict blue-green deployment.

[ Parent | on HN ]

Re:Rapid release at massive scale by stingraycharles on Friday September 01, 2017 @04:57AM (#15147073)

This is what we decided to do as well at my work. We've been using LaunchDarkly to handle the feature flagging, and we slowly migrate features from 0% to 100%, or first roll out features to less important customers, etc.

Works like a charm, our product owner and engineering team are happier than ever, since it takes away a big part of the risks and makes rollbacks a lot faster.

[ Parent | on HN ]

Re:Rapid release at massive scale by zatkin on Friday September 01, 2017 @04:54AM (#15147063)

How are they able to achieve slicing their userbase into thousands of pieces to do this gradual release?

[ Parent | on HN ]

Re:Rapid release at massive scale by keypusher on Friday September 01, 2017 @05:04AM (#15147091)

One way to accomplish this would be reverse proxy layer with sticky sessions that routs some percentage of traffic to X and some to Y.

[ Parent | on HN ]

Re:Rapid release at massive scale by mccv on Friday September 01, 2017 @11:54AM (#15149566)

This is in fact exactly what we (https://www.turbinelabs.io) do. With enough proxy work you can tee some traffic to version Y and observe success rate/latency, then discard the response (or compare it with version X's response), at least for idempotent requests.

[ Parent | on HN ]

Re:Rapid release at massive scale by sweetdreamerit on Friday September 01, 2017 @06:04AM (#15147256)

A sort of A/B test, where what is tested is the implementation.

[ Parent | on HN ]

Re:Rapid release at massive scale by user5994461 on Friday September 01, 2017 @05:24AM (#15147141)

>>> Where release engineering gets interesting is how you handle the happy path vs. a breaking release.

It's simple. Never make a breaking release.

[ Parent | on HN ]

Re:Rapid release at massive scale by oblio on Friday September 01, 2017 @06:52AM (#15147408)

This is basically a: "We kill the Batman" statement, on its own :)

[ Parent | on HN ]

Re:Rapid release at massive scale by user5994461 on Friday September 01, 2017 @10:00AM (#15148345)

On the opposite, it's more of a "dont kill the batman".

How hard can it be to not do something? Not very difficult.

[ Parent | on HN ]

Re:Rapid release at massive scale by sciurus on Friday September 01, 2017 @10:27AM (#15148558)

But you are doing something, writing then releasing software, regardless. It's silly to claim that "not making mistakes" is easier than "making mistakes".

[ Parent | on HN ]

Re:Rapid release at massive scale by dasil003 on Friday September 01, 2017 @10:47AM (#15148742)

Yes, it's very easy if you never release.

[ Parent | on HN ]

Re:Rapid release at massive scale by paulddraper on Saturday September 02, 2017 @10:59AM (#15156279)

You're referring now to "Don't commit any crimes"

[ Parent | on HN ]