Google Compute Engine Incident #16015

Re:Google Compute Engine Incident #16015 by jread on Tuesday August 09, 2016 @11:08AM (#12255028)

According to network checks we have running against VMs in each GCE region, this event resulted in about 1.6 hours of concurrent ICMP timeouts for every region (except us-west1 for only 10 minutes). We use Panopta for monitoring which verifies outages from multiple external network paths. When outages trigger we also use Ripe Atlas to confirm them using 100s of last mile network paths of which 85-95% resulted in timeouts. This is the second global GCE networking outage this year. These outages are particularly problematic because even multi-region load balancing will not avert downtime.

https://cloudharmony.com/status-for-google

Prior global outage - April 11: https://status.cloud.google.com/incident/compute/16007

Disclaimer: I am the founder of CloudHarmony Edit: Outages triggered due to ICMP timeouts

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by sshykes on Tuesday August 09, 2016 @11:20AM (#12255154)

Not accusing you of being intentionally misleading, but it would be nice if you put a clear disclaimer pointing out that you work for CloudHarmony.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by jread on Tuesday August 09, 2016 @11:21AM (#12255163)

I founded CloudHarmony and now work for Garter which acquired CloudHarmony in 2015.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by simonebrunozzi on Tuesday August 09, 2016 @11:30AM (#12255249)

I think it would still be nice to mention that you work there.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by lima on Tuesday August 09, 2016 @11:39AM (#12255330)

Their post-mortem states that only traffic for protocols other than TCP and UDP was dropped. Does your monitoring take this into account?

It states "1.67 hours" of downtime, which is not the same as increased latency due to sub-optimal routing.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by jread on Tuesday August 09, 2016 @11:56AM (#12255451)

Our outages were triggered as a result of ICMP timeouts.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by lima on Tuesday August 09, 2016 @12:42PM (#12255857)

In this case it would be fair to note that this outage did not affect all connectivity (web traffic would be unaffected while ICMP were blocked).

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by jread on Tuesday August 09, 2016 @11:23AM (#12255185)

Is there a reason for the downvotes?

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by slau on Tuesday August 09, 2016 @11:30AM (#12255250)

Most probably because you didn't edit to post a disclaimer. It just looks like someone trying to profiteer off of GCE's incident.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by jread on Tuesday August 09, 2016 @12:19PM (#12255653)

I've added a disclaimer. We do not profit in any way from these incidents.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by GilbertErik on Tuesday August 09, 2016 @01:26PM (#12256202)

While it's true that an auto insurance company doesn't profit off car crashes... if people _never_ had car crashes, we might not need auto insurance companies.

[ Parent | on HN ]

Re:Google Compute Engine Incident #16015 by ajkjk on Tuesday August 09, 2016 @11:30AM (#12255261)

Presumably because your post feels like an ad.

[ Parent | on HN ]