Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by ggthrowaway2020 on Thursday December 10, 2020 @04:19PM (#25378858)

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution.

In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying.

We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill? Google's alert was 6 hours late (according to them, this was root-caused to us, because we were submitting jobs continuously). They pointed to their TOS and said they don't guarantee on-time delivery of the alert.

I had to write up a blog post with fancy graphs and prepare it for social media before they finally agreed to eat the bill.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by dathinab on Thursday December 10, 2020 @07:28PM (#25380751)

(By now) GCP documentation says billing alerts can be late by days. Yes days not just hours. Totally crazy.

EDIT:

Link (see blue info box a a bit below the anchor on which the page is opened):

https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by donmcronald on Thursday December 10, 2020 @09:14PM (#25381453)

> Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays.

But if 1,000,000% lower doesn't work ($7 vs $70k) then...?

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by sandGorgon on Friday December 11, 2020 @01:18AM (#25382900)

P.S. so im not a googler.

you misunderstand the intent of this - you basically set this. even if it fails (because messages are delayed), google will refund.

This has happened to us before - they do a refund - since you had set the limits correctly. In general, they are not super assholes. I actually dont know a case, where they have refused to refund.

AWS is better here - since GCP doesnt have a support dashboard. So the "chasing them" experience is much worse.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by FeistySkink on Thursday December 10, 2020 @05:36PM (#25379730)

Is there a public postmortem anywhere? Your message points to 'no', but just in case.

[ Parent | on HN ]