Re:Ask HN: YouTube down?
by SteveNuts on Tuesday October 16, 2018 @10:17PM
(#18235477)
I thought they had all sorts of crazy canary deployment processes that will automatically stop/rollback the deployment if the failure metrics start increasing?
[ Parent
| on HN ]
There was a GCP Global network outage maybe a year or two ago, where their canary of a canary (Yo dawg, I heard you like bugs, so I I put a but inside of your bug) caused a cascading network failure.
It was a bad config push -- brownouts fighting with brown outs. Consistency issues. I bet this is happening.
Ran out of canaries?
Well, could be an unforeseeable-wack-crazy-bug. Can't plan for every possible failure.