Re:How release canaries can save your bacon
by londons_explore on Tuesday April 04, 2017 @06:33AM
(#14031159)
Not mentioned in this blog post is rollout related outages.
It is common for a system to work fine before and after a rollout, but during a rollout clients experience errors.
One might imagine downloading a big file for example, which takes an hour. If you are downloading that from http-server-v1, which is being upgraded to http-server-v2, there will be a small grace period for clients of v1 to complete their operations. That grace period in many datacenters is often 30 seconds. That means if your operation is long running, you would see a failure. The error code is usually HTTP 503, for which your client logic should retry the request with an exponential backoff.
If your client doesn't retry/resume the request, now you see the service as down, when in fact the error you are seeing is by design. It will happen for every release, but also when servers come and go for maintanance, or for a bunch of other reasons.
Good libraries will handle retries for you, but some don't properly, and thats a bug.
It is common for a system to work fine before and after a rollout, but during a rollout clients experience errors.
One might imagine downloading a big file for example, which takes an hour. If you are downloading that from http-server-v1, which is being upgraded to http-server-v2, there will be a small grace period for clients of v1 to complete their operations. That grace period in many datacenters is often 30 seconds. That means if your operation is long running, you would see a failure. The error code is usually HTTP 503, for which your client logic should retry the request with an exponential backoff.
If your client doesn't retry/resume the request, now you see the service as down, when in fact the error you are seeing is by design. It will happen for every release, but also when servers come and go for maintanance, or for a bunch of other reasons.
Good libraries will handle retries for you, but some don't properly, and thats a bug.