News for hackers. Stuff that matters.
Re:How not to get a $30k bill from Firebase by paulgb on Monday May 27, 2019 @05:12AM (#20020592)
I really wish cloud providers would allow users to set a hard budget that just stops the service if you exceed a threshold. I got a surprise bill from AWS this month (fortunately orders of magnitude less than this, but still ~15x my usual) and am thinking of moving from Lambda to a VPS just so the possibility of this doesn't keep me up at night.
Re:How not to get a $30k bill from Firebase by OJFord on Monday May 27, 2019 @08:05AM (#20021441)
I use VPSs for personal use for exactly this reason - the bill's the same (small amount) every month, and if load spikes I'm happier not being able to handle it than handling it for a surprise Xx bill.
Re:How not to get a $30k bill from Firebase by vertis on Monday May 27, 2019 @09:46AM (#20022148)
A few years ago I got a ~$3500 Amazon bill when someone broke into my account (through keys I'd inadvertently added to GitHub)[1].

I had no expectation of that money being waived, but Amazon did anyway.

Since then I've had billing alerts turned on at about the $150 mark (which is about double what my personal monthly Amazon bill is).

[1]: https://news.ycombinator.com/item?id=6911908

Re:How not to get a $30k bill from Firebase by hoaxcracker on Tuesday May 28, 2019 @04:13AM (#20028176)
Shameless plug: I know this plug should not be here but I think I can help some people with cost problem via (the easiest automation platform for AWS)[1].

I have a startup plan where you can run 60 executions in a month free of cost. Just reach out to me via https://totalcloud.io/solutions.html and refer YComb.

[1]: https://totalcloud.io

Re:How not to get a $30k bill from Firebase by grepthisab on Monday May 27, 2019 @07:42AM (#20021313)
Same here, with AWS specifically. I'm used to AWS so generally stay in that ecosystem. Designed a web app and backed part of it by lambda and api gateway. Via a surge in traffic, my monthly bill went from about $6/mo to $150 for a month. Didn't catch it until I got the bill for the month, the timing of which meant my next month's bill was about $80.
Re:How not to get a $30k bill from Firebase by dvfjsdhgfv on Monday May 27, 2019 @07:01AM (#20021111)
Only Amazon is adamant at refusing this, Microsoft already made it possible some years ago, and I believe Google too.
Re:How not to get a $30k bill from Firebase by donmcronald on Monday May 27, 2019 @07:37AM (#20021288)
Azure doesn’t have it for pay as you go.

https://feedback.azure.com/forums/906772-cost-management/sug...

A 6+ year wait for such a reasonable feature sucks.

Re:How not to get a $30k bill from Firebase by GFischer on Monday May 27, 2019 @09:55AM (#20022221)
And they have it for almost all other plans, so the technology is there.

It sounds like an intentional product decision and not a technical obstacle.

Re:How not to get a $30k bill from Firebase by stickfigure on Monday May 27, 2019 @05:46AM (#20020747)
You can? I have daily limits for my Google App Engine apps.
Re:How not to get a $30k bill from Firebase by inlined on Monday May 27, 2019 @09:57AM (#20022236)
Firebase also has a fixed price plan that can shut off your app if it blows through quotas. It’s called “Flame”
Re:How not to get a $30k bill from Firebase by EugeneOZ on Monday May 27, 2019 @07:44AM (#20021322)
Don't forget they are only warnings, they will not shut down your app and their update rate is not as frequent as you might expect.
Re:How not to get a $30k bill from Firebase by asciimike on Monday May 27, 2019 @10:17AM (#20022374)
App Engine PM: the GAE Daily Spend limits are hard caps that will shut down services when hit. They are different from the broader GCP Billing Alerts which are simply notifications.

The issue is that they only apply to certain GAE services (compute, legacy APIs, etc.) and not across the platform.

Re:How not to get a $30k bill from Firebase by EugeneOZ on Monday May 27, 2019 @11:41AM (#20022906)
Thank you for correction!
Re:How not to get a $30k bill from Firebase by inlined on Monday May 27, 2019 @09:56AM (#20022228)
If the concern is DB operations adding up you can lower your quota as well which will be strongly enforced and shut down your app.
Re:How not to get a $30k bill from Firebase by cavisne on Monday May 27, 2019 @05:13PM (#20025289)
You can limit the max concurrency of lambda functions which can help a bit here.

Imo the reason is this is for business/enterprise use cases configuring this wrong could be catastrophic, for side project/personal use cases its pretty well known you get 1 chance for a refund from the public clouds, after some manual checking to make sure you aren't trying to abuse the service.

Re:How not to get a $30k bill from Firebase by GFischer on Monday May 27, 2019 @09:53AM (#20022203)
Microsoft Azure has such a feature (although not for Enterprise users, which is evil), I´m surprised that AWS doesn´t.

https://azure.microsoft.com/en-us/support/legal/offer-detail...

Re:How not to get a $30k bill from Firebase by solidasparagus on Monday May 27, 2019 @02:22PM (#20024195)
You can set budget alerts, which aren’t as good, but they can let you know if the bill’s going off the tracks. I set up alerts at 50% and 75% of what I’m willing to pay and that’s gotten rid of my ‘unexpected bill’ stress.
Re:How not to get a $30k bill from Firebase by jsty on Monday May 27, 2019 @06:36AM (#20020984)
I think there'd be two big issues - (1) what is reasonable behaviour when you reach your budget, and (2) the consistency of billing.

On (1), if you get to say the 14th of the month and run out of budget, what would be the expected behaviour? - shut down all your services? delete all your data so that's not incurring a charge too? It'd probably have to be configurable on a service by service basis, and that's an awful lot of complexity to introduce for a tiny minority of their revenue base.

On (2), their billing systems are almost certainly eventually consistent in logging and charging usage, so they'd either have to shut you down early in anticipation of delayed charges, or let any delayed charges bring you over budget. Either is liable to make customers fairly unhappy one way or the other.

The TL;DR version - it'd be nigh on impossible to have good UX with such a feature, would be complicated to implement, and it's not likely to move the needle on any metric they really care about considerably.

Re:How not to get a $30k bill from Firebase by kkarakk on Monday May 27, 2019 @06:39AM (#20021005)
> it's not likely to move the needle on any metric they really care about considerably.

what about devs too scared to put their CC no in the till coz they've heard horror stories about being overcharged for mistakes? Just recently had to sort out a feature that a junior dev created using openstreetmaps coz they didn't want to be on the hook by putting their card into google maps... It was for an internal portal, usage wasn't even high enough to hit the monthly free tier!

Hour 1 experience also matters.

Re:How not to get a $30k bill from Firebase by p0llard on Monday May 27, 2019 @06:45AM (#20021029)
Why would a junior dev be using _their_ card details? Surely your company has a corporate billing account with Google for this purpose?

Perhaps I've misunderstood the situation, but if I were working as a dev at a company and was expected to provide my own card details to pay for APIs, I would walk out the door.

Re:How not to get a $30k bill from Firebase by kkarakk on Monday May 27, 2019 @06:47AM (#20021043)
They were just told to make the portal as a learning teambuilding exercise, choice of tools was entirely up to them - i even told the person company would pony up for anything they ended up spending. HR was being a pain about getting card details(it's not easy getting that stuff in india)

The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.

Re:How not to get a $30k bill from Firebase by hvidgaard on Monday May 27, 2019 @07:51AM (#20021352)
> The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.

As he should. Unless you have in written form that the company will cover the exact expense, then no one should use a personal CC for work use.

Re:How not to get a $30k bill from Firebase by kkarakk on Tuesday May 28, 2019 @04:50AM (#20028348)
Not everyone is a rockstar engineer in a top 5 company my guy, people have to start somewhere and paperwork is a huge show stopper in india in general.

from your comment history i see you're european.Europeans have way more protections and assurances in cases like this, india is a very different business environment...

Re:How not to get a $30k bill from Firebase by yjftsjthsd-h on Monday May 27, 2019 @11:22AM (#20022759)
> The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.

Sounds like a happy ending.

Re:How not to get a $30k bill from Firebase by dvfjsdhgfv on Monday May 27, 2019 @07:03AM (#20021124)
Was the choice of OSM a bad one?
Re:How not to get a $30k bill from Firebase by kkarakk on Tuesday May 28, 2019 @04:48AM (#20028340)
Seeing as how i had to learn all the GIS jargon OSM uses by default, it was bad for me. It's not a bad choice but since we as a company can afford the easy of Google maps it's a waste of time
Re:How not to get a $30k bill from Firebase by 7ewis on Monday May 27, 2019 @09:32AM (#20022032)
How did you manage to increase your bill 15x in Lambda?
Re:How not to get a $30k bill from Firebase by paulgb on Monday May 27, 2019 @04:51PM (#20025163)
It seems to have been a combination of a traffic surge and a third-party API being temporarily slow. The slow API meant that I ended up paying for more idle CPU time during the request round-trip.
Re:How not to get a $30k bill from Firebase by StreamBright on Monday May 27, 2019 @07:17AM (#20021191)
I guess you had not setup any budget alarms.
Re:How not to get a $30k bill from Firebase by paulgb on Monday May 27, 2019 @07:36AM (#20021275)
The alarms are why it wasn't higher than 15x.
Re:How not to get a $30k bill from Firebase by StreamBright on Monday May 27, 2019 @09:14AM (#20021912)
At what percentage you had the alarm then?
Re:How not to get a $30k bill from Firebase by tcgv on Monday May 27, 2019 @07:09AM (#20021156)
Amazon provides Billing Alarms for monitoring charges, which is very useful for anticipating unplanned cost surges:

- https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Re:How not to get a $30k bill from Firebase by donmcronald on Monday May 27, 2019 @07:43AM (#20021320)
Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.
Re:How not to get a $30k bill from Firebase by gtsteve on Monday May 27, 2019 @09:25AM (#20021993)
I guess you could have the SNS topic invoke a lambda that shuts down the specific service if you wanted that but I agree a budgets feature would be nice, especially when provisioning test accounts for developers.
Re:How not to get a $30k bill from Firebase by crankylinuxuser on Monday May 27, 2019 @10:24AM (#20022409)
Fine and dandy, but you can't use Lambda in fedramp.

You can't use most aws things in fedramp scope.

Re:How not to get a $30k bill from Firebase by brianwawok on Monday May 27, 2019 @09:43AM (#20022119)
On the flip side I would never want my real business to shut off. Hobby project yes! Real business no,
Re:How not to get a $30k bill from Firebase by donmcronald on Monday May 27, 2019 @04:34PM (#20025079)
AFAIK it's already considered a best practice to set up separate accounts for development and production (https://blog.codeship.com/separate-aws-production-and-develo...). Having a hard cap on an account dedicated to development would help a lot IMO.
Re:How not to get a $30k bill from Firebase by faster on Monday May 27, 2019 @12:04PM (#20023088)
Assuming your revenue and your AWS bill are strongly correlated, I agree. However, when a free service or a bug can increase costs by orders of magnitude without a corresponding increase in revenue your choices are to eat it (not sustainable, for most of us), throttle, or stop. Throttling is hard, so stopping is often the best choice.
Re:How not to get a $30k bill from Firebase by brianwawok on Monday May 27, 2019 @10:03PM (#20026625)
Disagree there, for a paid business.

If my business had 100k in revenue per month. Normal cloud spend is 2k per month.. some glitch made spend go to 2k a day. That sucks, and I hope I have good alerts to catch it within a day or two.. but telling my customers "screw you, bill messed up" for however long it took me to sort out the bug? I would not do that.

Re:How not to get a $30k bill from Firebase by influx on Monday May 27, 2019 @10:48AM (#20022556)
Sounds like the oomkiller. Sounds good in theory until it starts shooting processes in the head that you didn’t intend.

What would you like Amazon to do for stateful services? Should they stop and delete EBS volumes? What about databases? Simply shut them down? What happens when you lose data or it doesn’t come back up?

Re:How not to get a $30k bill from Firebase by donmcronald on Monday May 27, 2019 @04:22PM (#20025012)
EBS volumes have a size, so there's an upper bound to the cost. Most of the storage is predictable, so if I try to allocate storage where the monthly cost of the raw storage is (ex) 10x my budget, I wouldn't have a problem with the request being denied.

For non-storage resources like EC2, network bandwidth, etc. I'd be fine with having a hard limit where everything just breaks, especially for stuff that's not production.

There could also be better, self managed quotas on resources. SES is a good example. AFAIK the quota is all or none across the entire account. IMO, it's not a good idea to give a user that needs to send (made up numbers) 1k emails per day credentials that can send 250k emails a day.

I have 3 AWS accounts. I don't keep anything in my main account. It's for billing only. I have a sub account for production that I try to keep pristine. I have a sub account for development and testing. It's the development account that scares me. I spend less than $50 per month. I'd rather have my whole development account de-allocated than get a bill for $1k.

Re:How not to get a $30k bill from Firebase by meddlepal on Monday May 27, 2019 @11:05AM (#20022668)
As with all things just having the option would be nice, but it does not mean it is useful or a good idea for everyone to use it.

For personal stuff I would much rather a kill switch than a nasty surprise. For _some_ businesses, that is probably the wrong decision though.