Post-Mortem for Google Compute Engine’s Global Outage on April 11

Re:Post-Mortem for Google Compute Engine’s Global Outage on ... by erichocean on Wednesday April 13, 2016 @03:33PM (#11491442)

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?!

I'm all for automation, but WTF? Insert even a semi-competent engineer in the loop to monitor the configuration change as it propagates around and the entire problem could have been addressed almost trivially, as the human engineers eventually decided to do.

[ Parent | on HN ]

Re:Post-Mortem for Google Compute Engine’s Global Outage on ... by dsl on Wednesday April 13, 2016 @07:17PM (#11493144)

First of all, BGP is core to Google's load balancing architecture. So within a single datacenter you probably have at least a few dozen devices down stream from each edge router.

Secondly, I'm seeing just shy of 500 individual prefixes, 282 directly connected peers (other networks), and a presence at over 100 physical internet exchanges, just for one of Google's four ASes.

Would you be able to read over that configuration and tell me if it has errors?

[ Parent | on HN ]

Re:Post-Mortem for Google Compute Engine’s Global Outage on ... by DanielDent on Wednesday April 13, 2016 @06:55PM (#11493034)

Any sufficiently large system quickly reaches a point where a human has difficulty tracking what the system should look like.

Google has at least tens of data center locations, each of which will have multiple physical failure domains.

There are also many discontiguous routes being announced at all of their network PoPs. They have substantially more PoPs than data centers.

It very quickly gets too much to reasonably expect people to be able to keep track of what the system should look like, let alone grasping what it does look like.

[ Parent | on HN ]

Re:Post-Mortem for Google Compute Engine’s Global Outage on ... by jvolkman on Wednesday April 13, 2016 @04:01PM (#11491689)

The human part of any process will also eventually fail, and it's much more difficult to fix human bugs. Better to shoot for full automation.

[ Parent | on HN ]

Re:Post-Mortem for Google Compute Engine’s Global Outage on ... by heisnotanalien on Wednesday April 13, 2016 @04:46PM (#11492059)

But a computer could do the same thing. It would be possible to alert on all the routes being blown away?

[ Parent | on HN ]