Re:Ask HN: YouTube down?
by trevyn on Tuesday October 16, 2018 @10:02PM
(#18235341)
My vote: Bad configuration got pushed.
[ Parent
| on HN ]
I thought they had all sorts of crazy canary deployment processes that will automatically stop/rollback the deployment if the failure metrics start increasing?
There was a GCP Global network outage maybe a year or two ago, where their canary of a canary (Yo dawg, I heard you like bugs, so I I put a but inside of your bug) caused a cascading network failure.
It was a bad config push -- brownouts fighting with brown outs. Consistency issues. I bet this is happening.
Ran out of canaries?
Well, could be an unforeseeable-wack-crazy-bug. Can't plan for every possible failure.