Honestly, incident management is hard. It is a different skill set to straight up SRE management, and all the training in the world won't get you ready. Believe me it's true in AWS too, true in a lot of places.

It's complicated, and the human factor is massive in an unpredicted scenario.

I'm an IM myself, and regularly have to make the call regarding status updates, and also seen how other companies do it.

One of my favourites was a big it infra company sending a notification about an outage 6 hours after the event, after we opened and resolved a ticket with them. Was actually pretty useful but made me laugh all the same

The time you mark the dashboard as “red” is when the clock starts ticking against your SLA and error budget.

That makes no sense. The dashboard is for current best-effort knowledge. SLA and budget are for actual outages.

Not true.

A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality.

Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now".

Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...

Properly investigating takes time. There are constant reports of something not working by customers so they need to confirm that it actually is internal and widespread first.

Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.

when soemthing this broadly painful happens at a company of Google's scale --typically you have something like a hundred people internally that all got paged.

There's no reason for service credits to tied be to the dashboard and not to determinations made after the investigation is over.

The SLA guarantees a certain level of no problems. The dashboard officially says there's a problem. This creates liability.

They should have pro-active monitoring system and update status page automatically when it detected disruption. Especially when you account to it's extremely hard to get human supports from Google because they automated it.

I wouldn't be surprised if, at Google's scale, something or other is disrupted all the time.

I think I'd prefer a status page that reflected that, or even no status page at all with an explanation, rather than a lying one, though.

Google has multiple internal alerting systems and people are aware of any disruption/delay/anomaly from the minute it happens.

The on-caller for the specific service decides on how to produce.

The status dashboard is something which will be (manually, yes by hand) updated by an operations employee, who is a couple of layers behind the actual SWE/SRE who gets the page.

The status dashboard is something which will be updated by an operations employee manually (yes by hand), who is a couple of layers behind the actual SWE/SRE who gets the page.

They are just doing a dry run for when they finally close shop on Google Cloud...