News for hackers. Stuff that matters.
Re:How we spent $30k in Firebase in less than 72 hours by itcmcgrath on Wednesday August 01, 2018 @09:01PM (#17668471)
Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.
Re:How we spent $30k in Firebase in less than 72 hours by alasdair_ on Wednesday August 01, 2018 @09:09PM (#17668509)
>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it.

This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).

Re:How we spent $30k in Firebase in less than 72 hours by itcmcgrath on Wednesday August 01, 2018 @09:20PM (#17668558)
Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review.

Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too).

Doing my kids dinner, so responses might be slightly delayed.

Re:How we spent $30k in Firebase in less than 72 hours by cheeze on Thursday August 02, 2018 @12:03AM (#17669161)
This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly.

Its a problem with most cloud providers, but Google seems to be notorious for it.

Re:How we spent $30k in Firebase in less than 72 hours by deadbunny on Thursday August 02, 2018 @07:50AM (#17670746)
In my experience none of those folks are paying for support.
Re:How we spent $30k in Firebase in less than 72 hours by welanes on Thursday August 02, 2018 @04:07AM (#17669950)
From my experience Firebase are different. They're now a part of Google but support has been excellent anytime I needed it - both before and after they were acquired.
Re:How we spent $30k in Firebase in less than 72 hours by Humdeee on Thursday August 02, 2018 @08:54AM (#17671029)
It's quite the phenomenon. Reaching out via social media to garner a company's attention when all other avenues have been exhausted... what a world we live in.

"Company ignoring you? Send out a tweet, that'll work!"

And it blows my mind that it actually does. It's very sad.

Re:How we spent $30k in Firebase in less than 72 hours by stickfigure on Thursday August 02, 2018 @02:02AM (#17669511)
This is a pretty common train of thought, but it's not really a question of deciding between "my bill will explode!" and "my app will go down!"

You just really have to put some serious thought into what your daily limits should be, and add some reasonable alerting to detect surges. The tools are there and they're not terribly hard to use. It just tends to be an afterthought for most developers because this doesn't look like a customer feature.

Without these sorts of automated scaled services, the traditional behavior is "your app goes down". This is a big improvement!

Re:How we spent $30k in Firebase in less than 72 hours by jliptzin on Thursday August 02, 2018 @09:15AM (#17671142)
No, if your app is failing in a way that a page view now costs $1.00 instead of $0.0001 for example then the application should go down.
Re:How we spent $30k in Firebase in less than 72 hours by onion2k on Thursday August 02, 2018 @01:52AM (#17669475)
Your app shouldn't just fail if an API is unavailable. You should code things to fail gracefully. In the case of a traffic spike replacing the front end with an email capture form and a message saying "things are really popular right now, how about we remind you about this tomorrow (with a discount code!)" works well.

This is hard for some things, but your startup failing because you didn't want to do it is much harder in the end.

Re:How we spent $30k in Firebase in less than 72 hours by asadjb on Thursday August 02, 2018 @03:03AM (#17669698)
Most applications have atleast 1 SPOF, the DB. Just because the DB is now the Firebase API doesn't change anything.

It would be very difficult to build products in a reasonable amount of time if everything has to be coded defensively. I can build my app quicker, and remain sane, if I assume that the DB will always be available, and just fail if the DB isn't there. Same for things like S3 (which I think had only 1 large scale failure in recent history), Redis, etc...

There are APIs which can be unavailable and you need to work around those. For me, these are mostly third party services that I don't control. But then again, I'm not building the next Netflix. I don't have enough engineers to build an app that works with a chaos monkey!

Not the best approach for all applications, but has been good enough for most projects I've worked on. Just my 2 cents.

Re:How we spent $30k in Firebase in less than 72 hours by benguild on Wednesday August 01, 2018 @09:32PM (#17668605)
Yeah, that would happen with the Google Maps API before. 150k request limit was not enough for a front-paging, and there was no rollover.
Re:How we spent $30k in Firebase in less than 72 hours by iamgopal on Wednesday August 01, 2018 @09:13PM (#17668518)
So basically we want to have cake and eat it too.
Re:How we spent $30k in Firebase in less than 72 hours by paranoidrobot on Wednesday August 01, 2018 @09:22PM (#17668564)
This seems like an overly cynical/snarky response.

It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.

Re:How we spent $30k in Firebase in less than 72 hours by badlucklottery on Wednesday August 01, 2018 @10:51PM (#17668933)
>that the billing and billing limits should also be able to respond similarly

How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do.

Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).

Re:How we spent $30k in Firebase in less than 72 hours by Dylan16807 on Thursday August 02, 2018 @12:23AM (#17669219)
The problem is that the user gets those alerts, tries to change the limit they set, and it doesn't work. That is definitely not as good as can be done.
Re:How we spent $30k in Firebase in less than 72 hours by ajmurmann on Wednesday August 01, 2018 @09:16PM (#17668533)
Or have it be adjustable within a reasonable time frame?
Re:How we spent $30k in Firebase in less than 72 hours by ShaneOG on Wednesday August 01, 2018 @09:31PM (#17668602)
A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.
Re:How we spent $30k in Firebase in less than 72 hours by magnetic on Thursday August 02, 2018 @01:36AM (#17669426)
How did you get your Startup credits?
Re:How we spent $30k in Firebase in less than 72 hours by ShaneOG on Thursday August 02, 2018 @10:23AM (#17671577)
My company was part of a partner acceleration program, however you can apply to be part of the Google Cloud for Startups program here[1], without needing to be in an accelerator.

[1] https://cloud.google.com/developers/startups/

Re:How we spent $30k in Firebase in less than 72 hours by magnetic on Thursday August 02, 2018 @02:27PM (#17674095)
Thanks guys. Appreciated
Re:How we spent $30k in Firebase in less than 72 hours by purerandomness on Thursday August 02, 2018 @07:45AM (#17670726)
Not OP, but when our startup was accepted into https://www.thefamily.co/ (which is basically a European YC), amongst other perks, we got a huge amount of AWS credits. I'm sure other incubators/accelerators do similar offers for GCE/AWS.
Re:How we spent $30k in Firebase in less than 72 hours by mickronome on Wednesday August 01, 2018 @09:34PM (#17668625)
I would very much like to have something like Google authenticator, but for billing. With the ability so set it as alarms on my phone (and coworkers), preferably with some smarts to detect short usage spikes.

In essence, settings and updating amount, rate and velocity ( speed of rate change ) caps on the fly.

Then I can set whatever tight limit I want to, and not worry about burning through too much cash because of some simple coding, or config error.

A bug almost cost us several tens of thousands of BigQuery costs when a dev accidentally repeated a big query every 5 seconds in an automated script, and while we still had budget warnings, it still cost us a fair bit of money. Even after this, I found it tricky to set/catch budgets for single services. I think I had to use stackdriver to be able to get any kind of warning.

It was in the ’blinking lights and sirens’- territory fast!

Re:How we spent $30k in Firebase in less than 72 hours by tkrupicka on Thursday August 02, 2018 @02:03AM (#17669515)
Hey! This might not be the best place to contact you, but my team is currently in crisis from a nasty Android Firestore bug. When the user's device switches networks it loses the real-time event listeners and in some cases doesn't re-establish until the app is reinstalled. We submitted a bug report and source for an app with the issue but haven't heard anything in a week. We love the product otherwise, but I thought I'd bring it to your attention.
Re:How we spent $30k in Firebase in less than 72 hours by puf on Thursday August 02, 2018 @02:41PM (#17674229)
Hey there. Thanks for letting us know. We unfortunately don't have a fix for this at the moment. We've seen reports on network connections not being reestablished, but have been unable to reproduce it so far. If you've filed a bug report, the best way to help us reproduce it is by adding details to the report. That then also gives our team a way to get back in touch with you once they're able to reproduce (and then fix) the problem. Thanks again for the report!
Re:How we spent $30k in Firebase in less than 72 hours by snowwolf on Thursday August 02, 2018 @09:42AM (#17671276)
How about having default conservative alerts built in for new accounts (not caps, just alerts). That way people who forget to set them will be reminded the first time they get one.

An account that goes from $0 spend to $30k in 72 hours should really trigger some kind of flag - even internally within google. What if they didn't have any kind of grant and weren't able to pay?

Re:How we spent $30k in Firebase in less than 72 hours by kylec on Thursday August 02, 2018 @12:23AM (#17669218)
Are the caps on by default? If not, they should be.