The incident with the UI (where we suggested using gcloud temporarily) was opened in https://status.cloud.google.com/incident/container-engine/18..., but then what sure looks to me like the same incident was closed in https://status.cloud.google.com/incident/container-engine/18....

My working assumption is that 18006 should have closed out 18005. But now it sounds like there's a different issue, which we're working to get to the bottom of.

The people tasked with fixing this aren't the ones providing the updates.

Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident.

And this is likely a major incident with significant customer impact.

The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.

Incident manager isn't public comms person.

The person updating that status dashboard may or may not be an engineer, the IM certainly is.

They should take a few mins out of their weekend to update the dashboard regardless. If my small 25 employee company can do it Google can do it.

See my other comment, to say what, exactly?

That's yes, it's still being investigated?

Yeah why not? A billion dollar cloud provider can't have one person communicating to customers that are facing multi day outage. That not updating is an option is absurd in that time range.

Yes precisely. At least with a new timestamp we will then know the status dashboard isn't also broken.

At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.

I think the broader point here is that they (whomever they is) don't think the situation calls for it. That's why they said the next update would be on Monday on Friday.

You may feel that's a bad decision, but I doubt that people are in a panic because they can't push out an update that would not be noticably different from the last one.

Just to clarify, what should this update contain?

>Just to clarify, what should this update contain?

"We're working on it", possibly an ETA for a fix or some details. Technically it's fluff but people are not machines and the update is for people. We like the feeling that people are working on a fix, that people care and that the end is in sight. It makes the situation less stressful and, as for why Google should care, less stressed engineers won't bad mouth Google as much after the fact.

Compare Amazon's responses during their S3 outage: https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ (toward the bottom)

What's the better communication plan: detailed, hourly updates or terse, one-line blog posts scattered across several days?

I think this comes down to the two outages being not at all similar. An s3 outage affects practically every Amazon customer. This affects a relatively small number of go customers.

And I'm confused about what was good about that response. That article is about how the s3 outage caused so many issues that Amazon couldn't update their status dashboard to inform users at all.

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

It's weekend, why wouldn't you take it off? It's just silly software.

A quick search suggests that silly software is used by dozens of companies classified as hospitals or healthcare.

Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down.

https://idatalabs.com/tech/products/kubernetes

> Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down.

If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.

A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.

I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!

Ah, I guess I missed it. My fault.

More to the point, why would you depend on Google for any critical infrastructure after this?

Because you tried to run your own infrastructure and it was so much worse.

Then in a few months from now, people will be saying the same things after an AWS or Azure outage.

Am I the only one that finds this slightly humorous?

This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to?

It's likely the fix is checked in and will start roll out on Monday.

Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.

>This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to?

Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is.

You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...

It's not just the UI though. Resources have been exhausted, or that's the error I'm getting, in a lot of regions for both my work and personal accounts (GKE and GCE). Someone on the GCP slack also said they were getting similar issues from Dataflow so it seems to be widespread across products.

They have whole book describing who fixes, provides updates, etc. Fun meditative reading while waiting for the outage to get fixed.

https://landing.google.com/sre/sre-book/chapters/managing-in...

Looks like this time Mary took the whole week off without telling Josephine :)