And 50ms away from restoring service.
The degree of care that needs to be put into a release is related to the potential damage a bad release can cause and the time to repair.
Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was).
If you're building NES cartridges, you should really get it right the first time, since there's no way to update. If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly.
Sure, but fast rollbacks are much easier than fast rollouts.
> Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was).
A uniform fast rollout is almost always worse, and a separate problem from data recovery or fast rollbacks. But I agree with this paragraph in principle.
> If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly.
When you're doing more than basic presentation of semi-dynamic content, most CSPs make multi-region deployment very affordable and easy. It's generally a failure of early engineering departments or an accumulation of technical debt that makes it otherwise in 2019. Too many folks assume it is something they should do later rather than get right to start.
Canaries and regional redundancy should be part of every product's basic launch checklist, imo. People love to say any kind of software engineering best practice is bad, because they love to pretend software engineering has no value. But for every flash UX trick or tracking feature someone ships providing value, that's completely overshadowed by being down when a user needs you. Reliability not only saves you time at the developer level by making features more predictable, but it means you need to worry slightly less about funnel optimization because your user attrition is lower.
(Seriously, how do people come with such facile counter-arguments?)
It's actually not hard at all to write fast software deployment. Many people do it accidentally. But to tie it directly to editors and stovepipe the infrastructure around testing, that seems to me like a great business model that any rational engineer interested in reliability should immediately reject unless it's coupled with open source, statistically valid canary tooling at the bare minimum.
What you actually want at that degree of universal knife-switching is rollbacks.
This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?"
Just because your tests/ci/etc. can't catch every single issue that could go wrong doesn't make them useless. Just because people die every single day in car accidents doesn't mean car's safety features are useless.
You're correct in that a CI should not in any way be considered a "perfect solution for outages", but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful.
Actually, your refutation is precisely what I'm trying to say! The GP implied that CI and UTs fix these problems. They don't. In the same sense that seatbelts and air bags may reduce the chance of a fatal accident, they do not replace safe driving, well-trained drivers and intelligent infrastructure design.
> but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful.
I didn't say otherwise. In fact I agreed those measures should be in place. They're just not sufficient. You will still face bad releases even if you do this, and so you shouldn't do fleetwide rollouts unless you just don't care about your system falling apart.
Please don't misrepresent my comment.
Yes, multiple times. It just didn't help with each and every outage, which is a different thing.
For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot.
Besides the concern about outages is totally unrelated to "speed of deployment".
Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server).
So then maybe the idea that a prompt global rollout is a bad one even if you have good tests and coverage.
> For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot.
I'm here to tell you, as someone who has been at this for a long time and someone who is actually doing SRE at Google at the outer layers: No. It matters. We do canaries because they let you find problems no reasonable person cold catch with tests. Tests themselves are only cone component of a chain of reliability best practices, starting at coding practice and expressive type systems with static checks and moving outwards to reploying the code carefully and with intentionality.
> Besides the concern about outages is totally unrelated to "speed of deployment".
This is obviously false. Slower rollouts of new software and canaries are a proven way to reduce the impact of a bad push. Once your system gets modestly complicated, it can be very difficult to spot a bad push with naive unit testing which (let's be real here) often just mocks out dependencies anyways.
> Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server).
Right, but very fast deployment to client machines is not hard. The CD model is just inherently dangerous unless it has automated safeguards against the complexities of even a modestly large production app.
Personally, if your staging environment is CD I think that's great (up until your org gets big enough that a broken build pisses off other devs). But to prod? I will resign before I let you do it. Everyone who does software at scale agrees it's not a good plan to do CD on prod. There are whole books about why you shouldn't do it.
Non sequitur. This confuses "speed of deployment" (which is a technical issue) with "slower rollouts" (which is a devops choice).
You can have 50ms (or, generally very fast) deployment and still have canaries, slow rollouts, incremental rollouts to smaller audiences, and so on. Those are orthogonal, choices.
The slowness of deployment that Dark claims to solve (and which is good for any project to solve) is about the non-essential slowness because of convoluted processes, too many moving parts, accidental complexity, and so on.
>Right, but very fast deployment to client machines is not hard.
You'd be surprised.
No, I wouldn't.
Perhaps one of the promotion strategies could involve a human controlled criteria.
I would like to understand their plans on this point a bit better as well.
I see immense power in switching a promotion strategy away from large units, like entire containers or versioned artifacts, to fine grained components, like versioned functions.
Do you see any advantages here for improving your platform's availability?
To answer, the whole system is based on feature flags (which are equivalent in this case to green-blue/canary releases). So yes, there is a massive amount of control for this; in fact, that's the whole point of the article.
You just didn't talk about the actual complexity of deployment. And that's your call, but as you can see from this thread there are people who think CI/CD to prod with no controls is a good call.
From my reading of the article, the way that Dark is intended to work is that the language and environment enforces that changes to code accessible in production are forbidden to be made without putting the change behind a feature flag. Feature flags can be written so that flag flips are rolled out gradually - to whitelisted users, percentage of users, etc. (I'd be interested in how sophisticated the set of requests enabled for the feature flags can be specified.) Writing a tool to instantly revert the feature flags to their state at a previous timestamp is straightforward in principle. And presumably there'd be no need to write separate configuration files in a pure Dark environment, so changes to configuration would also be feature flagged. In this case, there would be in theory no need for a separate canary process, since the feature flags themselves can be canaried. I just hope that Dark will encourage teams to canary changes rather than have "easy" defaults of "cowboy deployment to prod".
Dark would probably make it very easy to roll back the simple changes that result in exceptions being thrown on every request. All the most interesting outages that I've seen have involved emergent interactions that put the system into a state unrecoverable without manual intervention. It's unclear how Dark would help with that from my reading.
A thing to clarify is that you're not really deploying code in 50ms. It's more that all the code you write is in production, behind a feature flag. But, you cant change unflagged code and you also cant change the old code, you can only set a flag that some users see the new code.
Knife-switch rollbacks are good though, tbf. I just don't think that's particularly challenging.
Why would I want that? How do I control it for green-blue? For rational isolation? Is there native support for canary releases?