Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) and you will have no way to fix it.
This bit me hard one day with appengine and lost us a ton of converting traffic, even though we tried to get the limit increased within ten minutes of the spike (and well before our limit was hit).
Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too).
Doing my kids dinner, so responses might be slightly delayed.
Its a problem with most cloud providers, but Google seems to be notorious for it.
"Company ignoring you? Send out a tweet, that'll work!"
And it blows my mind that it actually does. It's very sad.
You just really have to put some serious thought into what your daily limits should be, and add some reasonable alerting to detect surges. The tools are there and they're not terribly hard to use. It just tends to be an afterthought for most developers because this doesn't look like a customer feature.
Without these sorts of automated scaled services, the traditional behavior is "your app goes down". This is a big improvement!
This is hard for some things, but your startup failing because you didn't want to do it is much harder in the end.
It would be very difficult to build products in a reasonable amount of time if everything has to be coded defensively. I can build my app quicker, and remain sane, if I assume that the DB will always be available, and just fail if the DB isn't there. Same for things like S3 (which I think had only 1 large scale failure in recent history), Redis, etc...
There are APIs which can be unavailable and you need to work around those. For me, these are mostly third party services that I don't control. But then again, I'm not building the next Netflix. I don't have enough engineers to build an app that works with a chaos monkey!
Not the best approach for all applications, but has been good enough for most projects I've worked on. Just my 2 cents.
It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.
How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do.
Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).
In essence, settings and updating amount, rate and velocity ( speed of rate change ) caps on the fly.
Then I can set whatever tight limit I want to, and not worry about burning through too much cash because of some simple coding, or config error.
A bug almost cost us several tens of thousands of BigQuery costs when a dev accidentally repeated a big query every 5 seconds in an automated script, and while we still had budget warnings, it still cost us a fair bit of money. Even after this, I found it tricky to set/catch budgets for single services. I think I had to use stackdriver to be able to get any kind of warning.
It was in the ’blinking lights and sirens’- territory fast!
An account that goes from $0 spend to $30k in 72 hours should really trigger some kind of flag - even internally within google. What if they didn't have any kind of grant and weren't able to pay?
It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does.
The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go higher even if they wanted to.
When we brought this up, they told us they were technically unlimited because you could shard your data into different DBs if you needed more connections, which is like saying all restaurants are all you can eat because you can keep buying more food.
That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.
There are some ways to limit writing using the rules engine but you’ll still get charged for failed writes. :)
Only real way to rate limit firebase that I know of is to put some sort of proxy service in between, but then you lose a lot of the advantage of using FB in the first place.
That’s why I prefer to just rent whole machines on AWS. If I accidentally have some code stuck in an infinite loop making some O(1) call, don’t charge me $10,000 for that when it costs you nothing. If it’s actually consuming electricity and significant resources, I’ll know quickly because my service will go down, as it should, not scale infinitely until my company is bankrupt.
It's endless bill monitoring and budget approval.
I'll stick to a flat rate DO droplet.
So, if the cost cap is defined as $n per day, once you deplete it you'll be down for the rest of the day (or if you take some manual action to increase the cap, if the cloud provider supports it).
This problem is a function of the granularity. Imagine a system that let you say:
"I want to spend max $n per second with a extra burst of $m per day/week"
You adjust your "$n" to match the throughput you'd have if you had a fixed size "pay what you provision" system and reserve $m for lucky events as landing on HN.
The amount of planning you have to do is similar to the traditional resource allocation, but with the benefit of paying less than provisioned if you're not getting all that traffic.
For more capacity based services, sure, but it's more likely to be down than just slow. When systems are run at their limit they rarely operate the way they did with a little less traffic.
When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down.
When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.
Caps are tough though. I can certainly understand a use case that would want a hard circuit breaker that just kills everything it can once it hits a certain threshold. Sort of; you presumably don't want everything on S3, for example, to be deleted.
On the other hand, moving up the scale of serious businesses, I can imagine it would be hard to specify circuit breakers (rather than just alerts) and you get into issues of terminating services that affect all sorts of other services across the entire account.
We had daily hard caps and budgets alerts, but it's still an area we can do better.
(Disclaimer: Product Manager for Cloud Firestore)
Perhaps cloud providers should have some sort of hard circuit breaker option (though it won't help for some things like storage) but it's probably not a priority as not a lot of businesses--their primary customers--would be OK with effectively hitting the power button for their entire cloud account if they exceed some dollar amount that someone or other configured a couple of years ago.
It's not firebase's fault you ran bad code and have a huge bill. They have bills to pay also. Think they can tell their vendors 'sorry we can pay you this week. A customer ran up a 30k bill and they can't pay it. So we can't pay you right now. but lol bad code right? '
I don't blame Firebase at all though -- great product.
I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met.
All it takes is a rogue line of code in an endless loop or something, and you are bankrupt.
Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for less than $100 a month on AWS?