News for hackers. Stuff that matters.

A comment on Google App Engine and Cloud Datastore Outages.

Re:Google App Engine and Cloud Datastore Outages by markdog12 on Monday November 11, 2019 @06:19AM (#21503860)
Our app is down. I can't even access any pages in Google Cloud Console. Timing out. Sometimes it completes showing all our clusters gone, then a timeout error. This is brutal. This isn't just GKE either...

Edit: Things are working for us now

Edit: Still getting timeouts and service unavailable

Edit: I'm getting 503 (service unavailable) from buckets, but nothing on the status page indicating there's any issue.

Edit: Seems our Cloud SQL instance was restarted as well

Edit: Multiple restarts of our production database

Edit: Dashboard finally updated to reflect growing # of services effected

Edit: This wasn't an "App Engine" incident. It was a very wide-ranging incident. Just change the title to "Google Cloud Incident" and be done with it

Edit: Things have seemed to stabilize for us

Was supposed to have today off with my family (Remembrance Day in Canada), but now I have to deal with support issues all day. Thanks Google!

[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by sethvargo on Monday November 11, 2019 @08:25AM (#21504629)
I’m sorry this incident is taking time away from your family. Our team is working to mitigate as quickly as possible. Initially the scope of the issue was unclear, but now the title and dashboard are updated, sorry about that.
[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by mritchie712 on Monday November 11, 2019 @08:26AM (#21504641)
I have been fucking with this all morning thinking it was something we did (we actually were changing some permissions last night / today), how about an email? There was no push of this info, I shouldn't have to opt in to that.
[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by numbsafari on Monday November 11, 2019 @09:58AM (#21505309)
(Google Customer, not Employee)

We have two slack channels here. One is where our internal monitoring agents post, as well as an RSS subscription to the GCP status board. We have a separate channel for less critical things (like GitHub).

Unfortunately, it does depend on GCP updating their dashboard. However, to date, we’ve only been impacted by one of these major outages. This morning, I saw the alert in our channel, but all our things were still operating (fortunately!).

YMMV, but I’ve found this very helpful.

[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by sethvargo on Monday November 11, 2019 @10:18AM (#21505475)
FWIW, we're exploring ways to make this better. We know it's a point of pain for customers. One of the challenges is that the "project owner" isn't always the right person to receive these kinds of alerts, since often that's someone in the finance department or central IT team. There are a few ideas being thrown around about more proactive notification right now.
[ Parent | on HN ]