News for hackers. Stuff that matters.
Re:Google Cloud Networking Incident Postmortem by exwiki on Thursday June 06, 2019 @07:40PM (#20120299)
Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund?

For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Re:Google Cloud Networking Incident Postmortem by zizee on Thursday June 06, 2019 @09:03PM (#20120716)
Cynical view: By making people jump through hoops to make the request, a lot of people will not bother.

Assuming they only refund the service costs for the hours of outage, only the largest of customers will be owed a refund that is greater than the cost of an employee chasing compiling the information requested.

For sake of argument, if you have a monthly bill of 10k (a reasonably sized operation), a 1 day outage will result in a refund of around $300, not a lot of money.

The real loss for a business this ^ size is lost business from a day long outage. Getting a refund to cover the hosting costs is peanuts.

Re:Google Cloud Networking Incident Postmortem by idunno246 on Thursday June 06, 2019 @09:13PM (#20120779)
for your example, one day would be about 3% of downtime. My understanding of their sla, for the services ive checked with an sla, a 3% downtime is a 25% credit for the month's total, or $2500, assuming its all sla spend.

In this outage's case you might be able to argue for a 10% credit on affected services for the month, figuring 3.5 hours down is 99.6% uptime.

but i still agree, it cost us way more in developer time and anxiety than our infra costs, and could have been even worse revenue impacting if we had gcp in that flow

Re:Google Cloud Networking Incident Postmortem by zizee on Thursday June 06, 2019 @10:41PM (#20121210)
Good point, I stand corrected/educated.

From GCP's top level SLA:

https://cloud.google.com/compute/sla

99.00% - < 99.99% - 10% off your monthly spend 95.00% - < 99.00% - 25% off your monthly spend < 95.00% - 50% off your monthly spend

Re:Google Cloud Networking Incident Postmortem by ohashi on Friday June 07, 2019 @07:40AM (#20123506)
<95%... that's catastrophically bad.
Re:Google Cloud Networking Incident Postmortem by chrisseaton on Thursday June 06, 2019 @10:36PM (#20121185)
> For sake of argument, if you have a monthly bill of 10k (a reasonably sized operation), a 1 day outage will result in a refund of around $300, not a lot of money.

Probably literally not worth your engineer's time to fill in the form for the refund.

Re:Google Cloud Networking Incident Postmortem by msbarnett on Thursday June 06, 2019 @07:48PM (#20120359)
They write the need for the customer to request it into the SLA, probably on the theory that a lot of customers won’t, which saves them money.
Re:Google Cloud Networking Incident Postmortem by oceliker on Thursday June 06, 2019 @11:46PM (#20121510)
Not directly GCS-related, but there was a big Youtube TV outage during the World Cup of last year (I think it was during semi-finals?). Google did apologize, but they only offered a free week of Youtube TV, which they implemented by charging me a week later than usual. I didn't feel compensated at all (it was a pretty important game that I missed!)
Re:Google Cloud Networking Incident Postmortem by illumin8 on Friday June 07, 2019 @12:17AM (#20121630)
Wow, what a dick move...
Re:Google Cloud Networking Incident Postmortem by chipperyman573 on Friday June 07, 2019 @12:13PM (#20126052)
What would you prefer they did instead?
Re:Google Cloud Networking Incident Postmortem by oceliker on Friday June 07, 2019 @08:12PM (#20129705)
Honestly, just the apology would be better. I felt I was being tricked. I end up paying each month anyway; it changed nothing for me.
Re:Google Cloud Networking Incident Postmortem by mehrdadn on Thursday June 06, 2019 @07:45PM (#20120330)
Probably because it seems to be in the SLA that the customer must notify Google? https://cloud.google.com/storage/sla

> "[Customer Must Request Financial Credit] In order to receive any of the Financial Credits described above, Customer must notify Google technical support within thirty days from the time Customer becomes eligible to receive a Financial Credit. Failure to comply with this requirement will forfeit Customer’s right to receive a Financial Credit."

Re:Google Cloud Networking Incident Postmortem by lucb1e on Friday June 07, 2019 @03:02AM (#20122270)
So the answer to why it is this way, is because they wrote it down this way..? I think the real question was why this decision was made, not whether they announced this.
Re:Google Cloud Networking Incident Postmortem by sbr464 on Thursday June 06, 2019 @07:45PM (#20120333)
I agree, but it’s pretty standard SLA verbiage (from the telco/bandwith provider days) to require the customer to request/register the SLA violation to benefit.
Re:Google Cloud Networking Incident Postmortem by vageli on Thursday June 06, 2019 @08:09PM (#20120463)
> I agree, but it’s pretty standard SLA verbiage (from the telco/bandwith provider days) to require the customer to request/register the SLA violation to benefit.

FiOS has proactively given me per-day refunds of service without notification on my part. Weird to me that Verizon acts better than Google in this case.

Re:Google Cloud Networking Incident Postmortem by macintux on Thursday June 06, 2019 @09:35PM (#20120885)
It’s easier to determine “this line was down thus everyone along the line was also down” than what Google is facing.
Re:Google Cloud Networking Incident Postmortem by vageli on Sunday June 09, 2019 @12:07AM (#20137056)
Google knows the affected regions and probably has very fine grained data around this. I mean, they can even tell you metrics about your instances, they don't have monitoring on their own infrastructure to determine impact radius?
Re:Google Cloud Networking Incident Postmortem by macintux on Sunday June 09, 2019 @09:57AM (#20139207)
It’s a vastly complex problem. What servers were impacted? For what percentage of the overall outage was each server impacted? During that time, was the server offline or simply slower than usual?

And the part that even Google can’t know, even if they somehow can assemble all of the above: did it matter? Not all servers are created equal.

Small wonder they’re letting customers drive their own reimbursement process.

Re:Google Cloud Networking Incident Postmortem by sbr464 on Thursday June 06, 2019 @08:59PM (#20120692)
Interesting, which kind of fios account? (residential/SMB/data center interconnect) That’s ideally how it should be!
Re:Google Cloud Networking Incident Postmortem by vageli on Saturday June 08, 2019 @03:38AM (#20131147)
Residential.
Re:Google Cloud Networking Incident Postmortem by jasonjei on Thursday June 06, 2019 @10:35PM (#20121182)
My question is if you had to pay for Google AdWords and your site was inaccessible due to GCloud outage, do you have recourse on SLA for paid clicks? Or is that money paid to Google AdWords lost?
Re:Google Cloud Networking Incident Postmortem by stanley on Thursday June 06, 2019 @11:47PM (#20121519)
In the same boat.

The icing on the cake was they disapproved one of the ads due to the destination URL not loading.. which was in itself surprising, because everything outside of the affected region was running fine.

Re:Google Cloud Networking Incident Postmortem by all_blue_chucks on Friday June 07, 2019 @01:21AM (#20121857)
Why do retailers use mail-in rebates instead of just lowering the price?
Re:Google Cloud Networking Incident Postmortem by baddox on Friday June 07, 2019 @03:34AM (#20122410)
It’s probably more than just “some buyers will forget or not bother to redeem the rebate.” There are other reasons, like price discrimination.
Re:Google Cloud Networking Incident Postmortem by pishpash on Friday June 07, 2019 @11:59AM (#20125896)
That's for price discrimination. This is more like the usual case of don't pay if you don't have to, though in Google's case it could well be that they don't care.
Re:Google Cloud Networking Incident Postmortem by partiallypro on Thursday June 06, 2019 @07:48PM (#20120354)
Microsoft refunded after their latest outage in South Central. Google might announce a refund later, though I did read on here that some of their outage was not covered by their SLA.
Re:Google Cloud Networking Incident Postmortem by parliament32 on Friday June 07, 2019 @02:31PM (#20127377)
Same reason mail-in rebates used to be a huge thing on physical products.

They're well aware that ~50% won't bother, so that $10 discount per unit effectively becomes a $5 discount.

Re:Google Cloud Networking Incident Postmortem by ernsheong on Thursday June 06, 2019 @11:02PM (#20121303)
Customer having to request for refund has been documented in their SLA, e.g. https://cloud.google.com/compute/sla

Having said that, if Google wants to delight customers, they should give a free tier bonus to all customers for a certain period, but such a thing cannot be fair to everyone.

Re:Google Cloud Networking Incident Postmortem by rhizome on Friday June 07, 2019 @01:23AM (#20121864)
>Having said that, if Google wants to delight customers

It'd never happen, "delight" is an Apple principle.

Re:Google Cloud Networking Incident Postmortem by gerdesj on Thursday June 06, 2019 @09:31PM (#20120859)
You get what you pay for.