Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by gonzo41 on Thursday December 10, 2020 @07:26AM (#25372932)

I wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars.

If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a raspberry pi on your desk.

But really the first lesson you should learn in any cloud setup is Billing Alarms :)

If you're doing ML or CV work then it's probably cheaper to build on the desktop and port to cloud once you understand what the workloads are.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by onion2k on Thursday December 10, 2020 @07:35AM (#25373006)

For AWS you get about $0.20 per 1 million requests on Lambda.

If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article.

But really the first lesson you should learn in any cloud setup is Billing Alarms

Alarms only tell you that something is going wrong. They don't stop it. If your mistake is costing $1000/minute and you're an hour away from a computer you have a very expensive problem.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by gonzo41 on Thursday December 10, 2020 @07:43AM (#25373077)

You can trigger events from alarms. And Lambda's only last 15 minutes. So still cheaper than 75K :D.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by welkie on Saturday December 12, 2020 @02:16PM (#25400677)

That's not a bad idea. You could set it up to delete all Lambdas (assuming you've got a CI/CD system capable of redeploying them quickly later) if the billing goes over. Of course, this may hurt you more because of the outage it would cause. Up to you really.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by jjk166 on Thursday December 10, 2020 @09:09AM (#25373821)

So you're taking code that you haven't validated locally to see what resources it uses, you're putting this up on the cloud to test it, then you are immediately going to the middle of nowhere without your laptop/phone/etc, and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by patrickaljord on Thursday December 10, 2020 @02:41PM (#25377616)

> and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?

This is HN, many of us are solo founders with no coworkers or employees. Also how could a "friend" pull the plug? If it was a physical server running in your house maybe, otherwise you can't really give them access to your AWS account with all your private clients data in there.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by jjk166 on Thursday December 10, 2020 @02:54PM (#25377765)

If you don't have anybody who can monitor your test, and you're not monitoring your test, why are you doing a test?

As for having a non-employee pull the plug, set up an IAM user with permission to access the test instance

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by WrtCdEvrydy on Thursday December 10, 2020 @03:51PM (#25378459)

> you're not monitoring your test, why are you doing a test?

Agile. Bringing you bankruptcy at the speed of cloud.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by jschwartzi on Thursday December 10, 2020 @03:07PM (#25377908)

If I'm the only developer on a project and I really need to get to market I might do just that. I sometimes do day hikes on weeknights so this is actually a likely scenario for me.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by jjk166 on Thursday December 10, 2020 @04:40PM (#25379095)

Do you go hiking alone without your phone? That seems dangerous.

And why would you start a test if you won't be there to see the results of the test? Seems more sensible to either leave after you've run the test or wait to do so until you get back.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by jschwartzi on Friday December 11, 2020 @02:50PM (#25390045)

If a test is going to take more than an hour then I'm not going to sit around after work waiting for it to finish.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by jschwartzi on Friday December 11, 2020 @11:28AM (#25387505)

Yeah pretty frequently. It's not dangerous at all. Maybe if I was climbing alone it would be.

[ Parent | on HN ]

Re:Burnt $72k testing Firebase and Cloud Run and almost went... by gonzo41 on Thursday December 10, 2020 @07:40AM (#25373057)

Just to expand on this. You can have a hard limit. For AWS, create a role/user that's essentially ~root like access. Make a lambda function that's triggered by a billing alert at your threshold to just turn off things from most expensive to least. So turn of the DB servers. So the apps error out and the users go away.

[ Parent | on HN ]