So you'd think this would be the case. I certainly did when I first joined. In fact, I actually proposed something even more extreme than your suggestion (use thompson sampling to control the rate of task restarts and rollbacks). But in practice such a continual canarying process isn't actually any better than a staged canary (like .1% -> 1% -> 10%).

Consider the types of issues you run into, there are

- Things that are painful, but not destructive (minor performance regressions)

- Things that are highly destructive (data gets deleted, major performance regressions, etc.)

The first you can handle being deployed to many people, so if you detect at 7.8% of your users instead of 10% doesn't much matter, you can run it at 10% forever without issue.

The second you'll detect on a smaller population because the changes are catastrophic.

>In reality, neither continual-canarying or google-canarying work well for detecting anomalous metrics because the rollout process itself causes applications to restart, which in itself changes their performance characteristics (cold caches, empty queues, first-use delay, jit optimization, etc.).

The problem here is that most of the time, you don't care about startup behavior, but steady-state behavior. Imagine that a new version introduces a memory leak, so the old behavior was to linearly increase to 1GB of memory over 1 hour and then level off, and the new behavior is to increase unbounded until eventually the system OOMs and restarts or whatever.

A "google style" canary would roll out to 1% of tasks or something, wait a few hours, and notice the difference and roll back. You might even experience 1% of tasks restating, but it's likely that the system can sustain that.

With the continuous canary, you'll release to 1 task per minute or whatever, and only be able to notice any change after you've pushed out to 60 tasks, and the change will likely only be detectable with any confidence once you have a notable difference in 120 or so. At that point, you've released to more than 1% of your tasks (either that, or it takes you 8 days to release a new version).

You can fix that by slowing the rate of releases, but now it takes you 8 weeks to release a new version.

Plus, even worse, you're now in a much more fail-open environment. With a 1% release, you can sustain in the kind of bad state if all of your qualification tooling fails. If you're continuously canarying though, you have to be much more careful to make sure that your tooling won't continue to push if the tooling itself is broken or getting unusual results. It's a more risky set up.

You seem to have thought about it in depth and my comments maybe missing some insight

A) Your argument is optimised for a certain type of failure which leaves you open to the others ( issues that are detected after high ratio deployment ) B) You can monitor just the jobs impacted by the canary deployment C) the progressions you discuss seem strawman like. You can have a doubling progression starting at 1 or 5, a tripling progression

D) the point about detecting at 7.5 and living at 10, what if it’s detectable at 20 and you’ve gone all out by then

Another thing to keep in mind is that you don't just want problems to be detectable: you want them to be diagnosable and fixable.

If you detect a problem with FooBar frontend at 1pm, the first question anyone's going to ask is, "What changed in FooBar frontend or any of its dependencies at or shortly before 1pm?"

If the answer is, "Bazz backend deployed at 12:50pm" then someone with no understanding of the innards of FooBar or Bazz can roll back Bazz and there's a pretty darn good chance they'll have fixed the problem quickly. This is a scalable approach to fixing things that works well in practice.

If the answer is, "Nothing just changed. Every layer of the system has been gradually changing all day" then your oncall person will need an in-depth understanding of every service involved in the system as well as creativity in guessing where to start looking. It can work, but it's going to be slower and more difficult and fewer people will have the system-specific knowledge and debugging skills do it well and quickly.

A) like I say, in practice issues that are only detected after large deployment are small. If they weren't small, you'd have been able to detect them earlier.

B) absolutely, but if a large subset of the jobs recently restarted you get noise and nonsense

C) indeed you can do exponential continuous rollouts, but then you are getting pretty close to a 3 or 4 stage where each stage is a 5x increase in size or whatever.

All excellent points. I'll just add that often times step functions observed in obscure dashboards have been major signposts in identifying arcane root causes... (hey, look what ELSE happened suddenly at about 4.34am!)

Those gradual rollouts exist, but sometimes you have changes that aren't so easy to deploy a fraction of a percent at a time.