By definition, I don't understand AWS, so figuring out how to turn it off was nearly impossible. AWS "support" didn't exist. Stackoverflow AWS geeks were in high dudgeon that I'd ask the question, "how do i disable this?" and would kill my question. Finally some kind soul did give me the trick to finding the last service to disable.

BTW the tutorial is absolutely useless. Just a thousand different incantations to repeat. No real understanding communicated. Felt more like an ad for myriad services.

0/10 would not recommend.

The reason those consultancy firms exist is because billing scales terribly. Once you’re a business using AWS you’d likely have a multitude of projects running across a multitude of departments which need to be billed to a multitude of different customers and internet cost centres. This all needs to be processed by an internal 3rd party financial system managed by non-technical people who wont even know what AWS stands for let alone what it does and how it works. In those situations the problem of billing becomes exponentially more difficult than a one person hobby project.

Billing alerts aren't good enough.

Consider a scenario where you're consulting on someone else's small- or medium-sized project and your bug costs the client a huge amount of money in the middle of the night. Now who pays? Say goodbye to your paycheck or reputation, even though it should have been preventable.

Another scenario: you launch a startup, and a bug empties the bank account and kills the company. If the solution is to just not use things like AWS and GCP (including Firebase, which has no billing cap) when you're getting started, why are they advertised that way?

You can also set alarms that warn you of projected usage.

> Now who pays? Say goodbye to your paycheck or reputation, even though it should have been preventable.

If it’s legitimate usage then I’m not really sure what you’re advocating; are you implying a service being suspended in the middle of the night because a hard spend limit has been hit is somehow better for your reputation?

Or maybe you’re suggesting that it’s not legitimate costs, in which case you’ve set AWS wrong to begin with and thus your reputation probably deserves to be queried.

> Another scenario: you launch a startup, and a bug empties the bank account and kills the company. If the solution is to just not use things like AWS and GCP (including Firebase, which has no billing cap) when you're getting started, why are they advertised that way?

No cloud service operates that way. In that situation you’ll almost always get charges refunded. Even in instances of gross negligence (which would be the case here since for the bank account to be emptied it means you’ve not been watching your spend for more than a month and no business should operate that way)

I do get the points you’re trying to make but I’ve been working with the cloud for some time and have seen plenty of horror stories, all of which were due to gross negligence and most of which were still refunded by AWS as a gesture of good will. They’d much rather have your repeat business than burn their users with bills that cannot be paid.

I’ve built solutions for customers who would prefer an unplanned outage over an unplanned $250k expenditure. Many customers think that I’m a dinosaur for saying it, but if there’s a 2-5 year expected lifetime for something, it’s almost always more cost efficient to use traditional colo or VPS.

Also, operationally it’s possible to have something more than all or nothing. Big companies usually have “Tier-0” services that must be up all costs. AWS is no stranger to complexity - this type of function doesn’t exist because it would cost them money. They probably make 9-figure money from obviously idle services.

AWS is like Lego. You’re supposed to build on it to create the behaviour you want

If you are out of money, all of your services needs to be destroyed immediately: including all of your database hard disk drives, because every single piece of infrastructure yields some regular cost.

It means that it is NOT going to be “a simple unplanned outage”, it’s going to be more akin to formatting your hard drive with all of your family photos on it.

Pretty certain AWS just finds it easier to sometimes write off costs rather than implement something so radical that customers may be even less happy afterwards.

And yet, we have horror stories of students and even experts being hit by surprise AWS bills.

Although I agree about usage - I never go anywhere near AWS unless someone else is paying for it.

> And yet, we have horror stories of students and even experts being hit by surprise AWS bills.

They obviously didn’t bother to manage it. But that doesn’t mean it’s not manageable. It takes all of 5 minutes to set up a budget alarm on CloudWatch. It was one of the first things I looked into doing when I set up my own AWS account years ago specifically because I didn’t know what I was doing back then and thus didn’t want any surprises. If I managed it then, then I find it hard to believe others cannot too.

That's $9/month until I disable that credit card, which I might do one of these days.

I thought they'd notice and disable my account after I successfully disputed the charge one time, but the bills just keep rolling in each month.

This might be a bit overreactive, but... I'm building an MVP for a SaaS app and I sure as heck am not going to host it on AWS.

Happy to help you if you need any assistance

Disclaimer: I don’t work for Amazon but I do work heavily in AWS.

It's been 5 years since I was responsible for anything AWS, but back then at least support was surprisingly good even when spending less than $100 a month. Emails would get answered and I could often get someone on the phone within 24 hours if I needed.

I share this sentiment after going through a lot of tech tutorials and onboardings. And it's not limited just to AWS. I find myself forgetting most of the information I was supposed to learn. I'm trying to be extra mindful and offer additional explanations when I'm writing procedural guides myself, but I still have a lot to improve.

It can be really frustrating and annoying when you can't hunt down where costs come from.

AWS and Azure both have pretty straightforward free tiers, but people end up accidentally racking up large bills anyway. A UI to see a list of running services, sorted by cost either doesn't exist or is not prominent enough.

It's slow, you can't open links in new tabs, one wrong click and you have navigate to the right page and wait for everything to refresh again, but you can get the list of billed items. Plus a "Other subscription charges" that contains some mystery costs not linked to anything.

Hard to believe that really is the case for a product targeting serious customers in 2021. Doesn’t that make a surprise bill still possible?

And it makes sense from a growth perspective: new products grow revenue, bad billing systems only annoy customers (but mostly invisibly).

The UI to see a list of running services, sorted by cost is precisely what Vantage offers (and more)

I have one that goes off monthly at about how much I expect to spend in a month.

No. They don't have caps because it's really hard to implement and because the negative press of an app going down because the cloud didn't scale is far worse than any received by surprise bills. Scaling is a large chunk of what you're paying for, after all.

This is quite obvious when you look at how lenient AWS is with retracting surprise bills. And the money perspective doesn't make sense, either: AWS is living on customers that have bills in the 5 figure range and up. The occasional 10$ from someone playing around aren't even a drop in the bucket.

So you might say, simply project the cost (with some magic) and prevent that from going over the limit. So, imagine, your app suddenly experiences a load peak and you need to scale up. However, adding a VM would increase that projection too much. Do you not scale, despite this possibily being a small load peak, and let the app go down? Or do you risk the situation described above? Doesn't matter, you'll get bad press either way.

And beyond that, you'd still need to project costs like traffic volume, which can vary extremely. Not to say anything about the technical difficulties of coordinating that billing information across hundreds of services in real time.

And even if you do all that, you still get bad press of the likes of "we forgot to remove our payment limit and it killed our app while being on the front page (and our alert did not trigger because we couldn't afford another mail)".

There's no way AWS (or any other cloud) is eating all these drawbacks just to have a limit. I bet it's orders of magnitude cheaper to just eat the occasional surprise bill.

Azure have the ability to stop everything at a specific limit - they choose not to make it available,

I’m willing to bet it’s the latter. If in a normal account, you’d then incurred $152.78 or $166.39, did the limit work? Would customers agree?

My cloud bills continue to change for several days past the end of the month (for legitimate calculations that come in for usage incurred during the month).

We couldn't technically make it stop exactly at $150.00, but only at $167.89 or whatever, so we are letting it run to $15k.

For catastrophic cases it doesn't matter. If it saves a person from an unexpected $15k bill then it works. Even for many businesses it would be ok to drop everything - I know some which can withstand being offline for a day, but not a $250k bill.

Two approaches:

  A) Hard limits: freeze the services immediately if your cap is reached, ideally by giving a heads up some time beforehand with predictions, if possible; this is what many VPS providers out there do for unpaid bills and such, which makes sense
  B) Courtesy: allow the services to keep working, but at a degraded performance level - that's what some of the other VPS providers out there do; for example, decrease disk performance, cap the CPU performance, limit the network speeds etc.; probably eventually also block writes, but don't delete data outright; any of the aforementioned should trigger monitoring alerts on the developers' side and Zabbix or another solution would alert them in minutes, as well as the vendor should also send e-mails about these measures either currently being put into place, or about to be put into place, so that the necessary actions can be taken

There's a difference between having the current capacity with a degraded performance during the spike and killing the entire app. You don't always need to scale up, depending on your failure modes. Having consistent service response times is overrated, as is needing to serve every single request without ever telling a small portion of your users that your service is experiencing high load - there should be solutions in place to deal with the backpressure and prevent data loss even under these circumstances anyways.

Unless you work in a Governmental organization or another critical piece of software for society, degraded performance is probably okay and no one feasibly cares or remembers even small outages - regardless of whether it's large sites, or small non profits or even side projects. Whereas if you do, then you probably have enough money to throw around for billing caps to not be relevant.

If you subscribe to those beliefs about always needing to be up and serve requests, however, then there's another option:

  C) Billing alerts: something that most of the providers out there already provide in some capacity, however in fairly bad ways; if AWS can bill you for Lambda functions on a 1ms basis, then there's no excuse for not receiving billing alerts the very instant when this spike first happens: https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-changes-duration-billing-granularity-from-100ms-to-1ms/

> Doesn't matter, you'll get bad press either way.

Bad press? As opposed to what, going broke and not being able to pay your rent because of unpredictably large bills with no way to limit them, just because your side project got popular on Reddit or Hacker News?

There's a world of difference between what's needed by corporations and what's feasible for private individuals, so for as long as there's a chance of such bills, i will not use Azure, AWS, GCP or any other platform like that.

Remember: these surprise bills will only be "eaten" by the larger providers based on their own goodwill. There's not much preventing them from banning you outright.

There are other providers that are far more reasonable in that regard for my needs: https://news.ycombinator.com/item?id=28639196

The reality is kortilla is the same person that if AWS deletes all their data when they hit their "cap" to stop the billing will be on here complaining.

Payment method not go through? Hit your cap? To stop charges AWS needs to delete forever almost everything in your account. All your S3 data gone. All your backups / databases and archives gone.

Oh - you actually DON'T want them to blow up your platform? Maybe they could provide instead a billing console

https://console.aws.amazon.com/billing/home

or maybe alerts and alarms?

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Cost or usage budgets (and many more)?

AWS Budgets (includes alarming options).

Some folks seem to have almost no clue about what AWS customers who pay the billions want. Is there ANY chance that AWS listens to its paying customers? Maybe has become successful by doing so (at the cost of total feature sprawl in my view?).

My quick trick is to login a day or two after I think things are shut down and look at projected bill and current month billing. I've left some very large instances running a time or two, easy to turn off.

And with billions of requests (PER SECOND) on the aws network, there is NO WAY they are doing real time billing. That is not happening. Look for daily aggregation and similar. Just the scale of permissioning on API calls must be insane per day. These are going to need to be doing local counters that aggregate periodically.

I do wish they'd maybe aggregate 4x per day (6 hours).

I’m not saying it would be simple, but it wouldn’t be a bad idea to have a global monthly billing maximum where it’ll nuke the account at that number. There’s lots of new developers that are probably too scared of the free tier to use it without something like that (I have AWS experience, and I don’t use it for personal stuff specifically because that doesn’t exist. Which means I probably won’t tell my employer to use it either).

I'm serious, what large businesses wants to lose EVERYTHING (all static IP's, all glacier and S3 data, all database and compute) over a billing issue.

Surely it's the customer's role to decide whether or not it is a bad idea?

How 'bout you run a non-profit and have an allocation to run the services, would you prefer your nonprofit to lose the website for a couple days, or for it to go bankrupt?

What if you run a company for which online presence is a means of advertising and not the revenue-generating platform, would you prefer to run your advertising campaigns on a budget or without limits?

And so on.

Realize that most customers are more focused on will their data be preserved.

Blowing out your entire EC2 / RDS / S3 / Glacier backup stack over a billing issue (or someone setting a cap up in the accounting department) makes no sense.

Are major customers really asking for this? Why risk it, why even build a tool that can blow out a customers setup so completely.

This is why I don't understand HN sometime saying AWS is "BS" etc. Does HN not thing AWS talks to their big customers to find out what they want?

Certainly it is the customer's decision to make, and not AWS' ?

is it?

if you gave user an ability to: scale infinitely or scale up to the $$ point, then how would anyone could be reasonably mad?

It's like asking why the knife you're using keeps cutting you when you put your finger on the sharp end. Learn to use your tools, and they won't surprise you. Learn to track your costs, and you won't have unexplainable recurring bills.

Most cloud providers don't have this billing problem and so the analogy breaks. Its more like, why does this knife keep shooting me in the foot?

I spent hour or two trying to find anything running and didn't find a damn thing. Yet Amazon decided to charge me a buck or two every month for "storage" so I just canceled whole account. I mean, if I can't find what I am paying for when not using it, how am I supposed to understand the bill when I actually have a dozen of instances?

In both cases I learnt my lesson. I'm careful not to put my fingers where a knife can cut it, and to not put my bank details where AWS can charge them.

100 of them have provided an opinion and 99 of the opinions are that it is. Everyone in the room is bleeding.

I think it's fair to say that there's an issue with the tool, Stockholm syndrome not withstanding.

You can even enable scans for unused respurces that generate costs..