Re:How Canary Deployments Work in Kubernetes, Istio and Linkerd
by shereadsthenews on Wednesday June 05, 2019 @04:36PM
(#20108861)
In practice it is easier to have same-sized experiment and control deployments than it is to normalize all the stats on a per-instance or per-request basis. I get more signal out of having identical populations than I have ever been able to achieve trying to normalize for known differences.
Re:How Canary Deployments Work in Kubernetes, Istio and Linkerd
by hinkley on Wednesday June 05, 2019 @04:42PM
(#20108918)
Some data collection tools allow you to add metadata to the metrics, so for aggregate alerts you can use all of the data and for things like canaries you filter based on the metadata.
If the canary is generating too much traffic to a service we need to know about that, and that will be a proportional alert. But at the same time we also need to know that the overall request rate to all servers combined has dropped precipitously and may mean we are having gross routing issues. In the rare event where someone is rolling out a huge improvement, like better cache infrastructure, you disable the alert, and possibly offer a new one.
Likewise, it complains that "10% canary is hard" because you need to scale the deployments in sync. This is a problem with your monitoring layer - Your alerts and triggers and thresholds should all be based on proportion of traffic, not the other way around. Your graphs should scale to the number of canary instances. Every canary metric should either be an absolute, or a proportion of traffic.
In short: Your canary should look as close to normal traffic as possible. There should be no appreciable differences between the traffic going to your canaries, and the traffic going to the rest of your production deployment. Your canary is a production deployment and any deviance needs to either be intentional or is a sign of defect.