Today’s outage for several Google services

Re:Today’s outage for several Google services by jjoe on Friday January 24, 2014 @09:35PM (#7119602)

Reads familiar? The outage pattern is identical to the one impacting GitHub January 8 [1] with my comment in the thread discussing the root cause [2]. Systems generating config files and then pushing them out to services within the infrastructure without proper checking and linting. In google's case, I just can't believe their systems are so delicately integrated and such critical component so botched up.

[1] https://github.com/blog/1759-dns-outage-post-mortem

[2] https://news.ycombinator.com/item?id=7081913

[ Parent | on HN ]

Re:Today’s outage for several Google services by magicalist on Saturday January 25, 2014 @03:19AM (#7120253)

Can you be more specific? They don't sound very similar at all, though to be fair, google didn't provide that many details here. They both involve generated configuration files?

> Systems generating config files and then pushing them out to services within the infrastructure without proper checking and linting.

It's not really that easy, as "proper checking and linting" might as well be phrased as "sufficiently smart checking and linting". You can have amazing checking and linting and still let a bug pass through.

[ Parent | on HN ]

Re:Today’s outage for several Google services by bigiain on Friday January 24, 2014 @10:59PM (#7119790)

Modern DevOps - the ability to deploy catastrophic mistakes globally and instantaneously!

(what could possibly go wrong?)

[ Parent | on HN ]

Re:Today’s outage for several Google services by rbanffy on Saturday January 25, 2014 @09:09AM (#7120801)

I once experienced a similar issue (on a much smaller scale, obviously). Our DNS zone files were generated by a programand, thanks to a subtle bug, we went offline. The catch was that the generated files were valid and no amount of linting would have saved us from syntactically perfect wrong data.

The checks need to be quite sophisticated. We patched the generator, but never got to implement a test deploy mechanism to ensure the config made sense before deployment to a real production environment.

[ Parent | on HN ]

Re:Today’s outage for several Google services by brianr on Friday January 24, 2014 @09:43PM (#7119628)

Also reminded me of this CloudFlare outage from last year, where a bad configuration rule caused their routers to crash: http://blog.cloudflare.com/todays-outage-post-mortem-82515

In that instance the rule was written by a human, but deployed automatically to their entire infrastructure before they could notice the problem it caused.

[ Parent | on HN ]

Re:Today’s outage for several Google services by antimora on Friday January 24, 2014 @09:51PM (#7119641)

Sounds like a problem what Dropbox had experienced recently too: https://tech.dropbox.com/2014/01/outage-post-mortem/

[ Parent | on HN ]

Re:Today’s outage for several Google services by rachelbythebay on Friday January 24, 2014 @10:07PM (#7119670)

I wonder if it was gen_aggregator.py again, 6 years later.

[ Parent | on HN ]