The problem isn't that it's too hard. It's that it isn't a business priority. As soon as it becomes prioritised, a legion of very smart AWS engineers will solve it.
Everything has trade-off's. When a responsible person says something is "too hard" it's because they've evaluated the consequences and have deemed them too costly in terms of time, resources, cost, maintenance, or strategy/lost-opportunity. Some things really are "too hard".
I agree it's not too hard technically but am surprised at your contradictory suggestion AWS would deploy "a legion of very smart engineers" to solve it if they decided to implement that ordinary feature. How smart are they?
I suspect the truth here is a bit more ugly. AWS doesn't want to do it because they see the pocket-change taken from accidental mishaps as a net positive. They don't care if it inflicts suffering on students and novices. Sure, they'll fix individual cases if they're shamed publicly or if the individual contacts them in exactly the right way, but at the end of the day, it's more money in their pocket. For every 200 dollar oopsie that gets charged back, how many go silently paid in humiliation and hardship?
What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.
I would guess most companies spending millions per month on AWS are just guessing what next month’s bill will be, and have been conditioned to not freak out if it’s a million over, as long as that doesn’t happen every month. That conditioning has to start early.
Why not? It would make companies happy.
The reality is that, as much as many would like this feature, I'm not sure that a lot of large customers are actually asking for it. Does anyone who works at a large company know if they've asked for this feature before?
I can imagine why startups might want it, but also, AWS is extremely forgiving with startups and throws tons of money at them, so idk.
what do you think will happen to your EC2 instances if your payment method on file with Amazon fails repeatedly and for more than 30 days? Do you think they will just keep those instances intact indefinitely without payment because they promised not to 'randomly destroy things'?
(Had to resist the urge to post “Occam’s Razor” or some other incorrect thing to bait someone to correct me)
As someone who tries to always get things right, it somehow seems wrong to deliberately post wrong answers.
I know it won't go over well at Stackoverflow, which is one of the very best places to get accurate and helpful information.
Now I'm positive that he was doing the same thing as you describe.
The effect on me was a reduced interaction with that person, I'd much rather have an honest answer or a "I don't have time" answer.
"Too hard" is not an excuse. They have all the resources they need at their disposal to get the job done.
A culture they feel aligned with.
They are poor or high burn rate consumers.
They are curious.
But, it would seem, most of all they have different priorities that you.
I write that as someone who routinely dismisses their recruitment attempts.
Let these people fix climate change then.
Sarcasm aside, it must be boring working in a place where no problem is ever too hard.
that's cheaper than the pandemic. no reason it couldn't have been done already.
i no longer consider climate change a hard problem, the only hard problem is building political momentum without getting shot
They don't. I've looked. They all have some ultra complex scheme for monitoring billing and programmatically shutting down services, which is a favorite recommendation by apologists, but none have a nice, solid "shut down everything" plan that's usable for learning and testing.
AWS does have billing actions which are fairly new, but they're still a bit too difficult to deal with if you also follow the advice for setting up a root account for billing plus sub-accounts / organizations for use.
Azure has been ignoring the request for over 8 years [1]. There's a warning they're moving away from User Voice to a product-by-product solution for feedback, so that request might disappear altogether. I reached out to them and politely asked for an update and they said they'd look into it, but it's been over a month and I don't think there's going to be an update without some pressure. If anyone has a good social media following and would like to see hard caps on spending in Azure, tweet @AzureSupport. Maybe if enough people do they'll actually follow up.
> PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.
I agree 100% with this because all I really want is something along the lines of the "personal learning" account that's described in the article. I want a dev / testing account where I can learn and test assumptions about billing without risking a massive overage that I wasn't expecting. They don't have to get into the complexities of dealing with production accounts.
AWS, Azure, and GCP have all had close to a decade to figure it out, so I think the only way it'll happen is if we start asking legislators to regulate those providers. It would be relatively simple in my opinion. Cloud infrastructure providers must allow users to indicate a maximum amount of spending per month and may not charge more than the indicated amount.
I think big tech is proving they can't self regulate, so it's time to quit letting them.
1. https://feedback.azure.com/forums/170030-signup-and-billing/...
They just refuse to implement this system for non dev licenses.
This is simply not true, to the best of my knowledge. GCP has budget limits, which can send alerts, but outside of manually catching a pubsub and shutting down billing on your own, there are no configurable hard billing limits.
I'm unsure what to make of the rest of your rant.
https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
I'm not sure if that's what you meant by "manually catching a pubsub" -- you have to manually set up the script once, but it will run automatically when triggered.
We go over our budget "limits" literally every month. We do this because they are simply alerts, nothing more. We have separate alerts set up for 25, 50, 75, and 100% of our "limit" so that we can track variable usage and catch spikes early.
We, of course, pay our full bill every month - over the "limit".
I've shared this before on hn, but I was about to bite the bullet on a fire base subscription to back a web app I'm building. Google announced its cancellation the week I was going to subscribe, I was hitting daily limits in my testing with my garage code and wanted the extra buffer to speed up my progress. As soon as they announced the cancellation I ripped out fire base and have been working on a swap to hasura/postgres/auth0. I can't afford, likely multiple, hundred or thousand dollar mistakes for a hobby.
"Just code better"
I'm trying...
Not all Azure subscriptions types are like this though.
Also documented at https://docs.microsoft.com/en-us/azure/cost-management-billi...
Make a SKU that's not for production with a $100 setup fee to prevent abuse and prepaid credits for consumption.
Basically you add a budget to your project, create a pub/sub channel, then make a lambda that subscribes to the channel and turns off billing when it receives an overbudget message to shut everything down.
Or you could integrate it into your project directly to handle this more gracefully.
You can even send a test message, to check that everything does indeed shut down, but the whole process is extremely clunky and error-prone.
If you have Visual Studio professional or enterprise license, then you can avail recurring credits, which are useful for learning
Enterprise deals can end up changing the story it seems
The best you can do is build your own service to monitor your bill and manually cut off your billing account if it gets too high. Hope you did it correctly because there's no good way to test it! And the bill notifications can be delayed an arbitrary amount of time with no guarantees, and disabling billing can take an arbitrary amount of time too, so you can still be screwed even if you did it correctly.
I think it's a glaring hole in cloud platforms. I get it, it's hard to implement in a way that is forgiving and doesn't cause a bunch of support tickets from people who set their limits too low and made their site go down. But that's why people use cloud platforms, so they don't have to implement the hard part. It's beyond me why the cloud platforms don't do something better here.
You're talking a 'get paid/don't get paid' decision here.
When you look at this on a high enough distributed level, across enough users, this is the profit margin. Enough people can screw up and then be able to pay the cloud provider, that it is part of the business model. You're required to run some chance of producing a legitimate megabill.
Why be forgiving when your business model requires that you set traps for fools?
No, they don’t, the article is misleading if not technically wrong (since they use “and/or” to link billing caps and better whole-project deletion, the latter of which mitigates a subset of the problem but doesn’t address accidental unbounded spend from a project that you want active at some level) and the same general class of complaint is raised for the other clouds on this issue; the frequency of the complaints seems to vary (ordinally, but not linearly) with usage of the various clouds.
The possibilities are endless. I personally don't see the IBM comparison.
Same with any technology, you should read first all the documentation that there is, watch all the videos that AWS releases every year for free where they explain in detail every service that they have, and try to explain the best practices of it - this is all for free, or you could spend $1000 for a course, instead of risking to lose $10.000.
If you jump in to the water without knowing how to swim, and then get angry at the water because you drown - well...
After 18 years of age you become a grown up, because your parents are not responsible for you anymore. You become responsible for yourself. That is what distinguish a child from a grown up.
Of course a parent needs also to tech you responsibility and what it means.
Mine for example never did, and I had to learn life the hard way. At first I was blaming others, but then I realized that to really grow up, I need to stop blaming others and start owning my mistakes.
So, it is not expected for anyone to know everything, but it is expected that if you want to learn something new, you need to first research the topic.
It also doesn't solve the problem. If the user makes a mistake they still lose a ton of money.
It's a really interesting problem. It's getting too big and too scary for a new developer to jump in and try AWS services, which means new devs will move on to newer services with better UX. I wonder if this sort of problem is one of the few existential threats AWS could actually face. Its possible it could end up as the Oracle or Salesforce of cloud services, very successful in enterprise but definitely not a good thing to have on your resume if you want to join a startup...
I don't believe there will be any new services with better billing UX through competitive pressure. You're basically asking the competing thing to take less money in order to win, as the distinguishing factor. It doesn't make sense as a way for them to win against the larger thing taking more money.
Probably the only practical solution would be arbitrary legislation forbidding the practice, as if it was anti-usury laws. Basically taking the guise of society and saying 'you may not do this, even if people are dumb'. And then you've got a problem (at scale) of people figuring out how to do hit-n-run megabilling and walking away under the legislation, having intentionally taken what they 'mistakenly' did. An 'oops, I accidentally a bitcoin' defense.
2. If money is a concern to you, then I should focus on learning how exactly the billing works and how to monitor correctly. This way you can build a product the right way, not to mention AWS by default has limits on their services set with limits that prevent you from doing something incorrectly. For example you can only make 5000 requests a sec on the API Gateway, you can only have 1000 concurrent lambdas, you can spin only 25 ec2 instances, ecc... (true, not all services have limits like this - but then again, if you want to use one, the first thing you should do is check the pricing page, this is what I do fro every new service that I'm planing to use).
3. AWS is not for developers, it is meant for SysAdmins and DevOps (true that some marketing materials are not clear on this), they should be the one configuring it to allow developers to host their code. If you want a turn key solution, then there are better solutions, like Heroku - incredibly easy to use and understand and have a much simpler billing structure.
With AWS you can do anything you want, AWS provides lego blocks, what do you build with it is up to your imaginations, and for sure it is not meant to be use directly by developers who have no idea how networks, computers, databases, cpu, ram, policies, storage etc works. Developers should focus on coding, and SysAdmins and DevOps should focus on managing the infrastructure.
And if you want to learn AWS because you want to be a SysAdmin, then it is true, that AWS could have a plan for beginners with even smaller default limits, and limits set on everything - this way you could more safely play with what they have. This would be a nice things to have in this case for sure.
But because they don't provide such thing, you need to be the responsible one, and start learning AWS the right way, and not get in gun blazing, and expect all will be ok. My recommendation is to learn one service at the time. If you do this, over the years the acquired knowledge will be gold. Plus the more services you learn the right way the easier it gets.
> studying years to be a doctor [...] Should you just start cutting people
Spoken like someone who’s never used AWS or been a doctor. Your analogy is horribly, badly flawed. Doctors do start cutting people, in the US nearly all med students dissect a cadaver at the start of their first year. What you’re suggesting is doing years of pure documentation reading, unguided by other people or a curriculum, before practicing AWS, which would be silly and a waste of time. People learn by practicing, which is why med students dissect cadavers, and which is why AWS offers a “free” platform to learn by practicing, tutorials to guide the learner, and advertising to attract learners.
> If money is a concern
Maybe this is why AWS offers a “Free Tier”? https://aws.amazon.com/free/
> AWS is not for developers
That’s not what AWS says https://aws.amazon.com/developer
People in this discussion are complaining that the free tire is misleading, plus not all services are covered. And it is true that if you turn on a bunch of server and you don't pay for a year, but forget about them, you will be charged the moment the year passes. Not to mention that you will be charged if you use the CPU of the free tire server to much - which probably very few people know about.
> That’s not what AWS says https://aws.amazon.com/developer
And my point is that the marketing of AWS is misleading, they try to convince you that if you don't know anything about computers, but you know code, you will be able to manage AWS. This is very misleading because AWS tries to make you think that AWS is a service like Heroku, simple to use, and there is just one button to push to make it all work. Completely false. I've seen countless AWS accounts that were completely misconfigured by developers who thought that AWS easy to manage. A basic example is the autoscaling of EC2. People will go to the autoscaling section of EC2 "enable it" and be superseded when it dose not work. Where the reality is that you have to do 8 other things to make it work, not to mention the work that needs to be done in the OS itself.
> Select a learning path for step-by-step tutorials to get you up and running in less than an hour.
Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
It is very dangerous. If you select the full-stack tutorial you get: "Time to Complete 30 minutes". It should say: "30 min to ruin your life" ;)
If you want to really learn AWS, then this page should be used as a reference of how to design a stack. If I were you I would read the tutorials to see which services are needed for a solution, but before doing anything, I would read the docs for each of those services to really understand them, then I would go back to the tutorial and actually do it, and - MOST IMPORTANTLY - I would read the pricing page for each service that you are going to use.
> Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
100% - when I started working with AWS in 2016 I had a very hard time figuring it out, because I was looking for the simplicity the the marketing team was writing about. I really don't like what the marketing team tries to tell you, because it dose not exist.
Regarding an approach to learn about AWS, I would start with all the serverless services that they have, since the pricing for most of them is ideal for beginners (WARNING - read the pricing page for each since not all have a free staring plane, like S3 and DynamoDB) and for simple weekend projects.
For example, I did build this project a while ago: https://github.com/0x4447/0x4447_product_s3_email, if you scroll down to the pricing section you will see this:
``` All resources deployed via this stack will potentially cost you money. But you'd have to do the following for this to happen:
- Invoke Lambdas over 1,000,000 times a month - Send and receive over 1000 emails a month - Perform over 10,000 Get and Put operations and over 2000 Delete operations in your S3 Bucket - Exceed 100 build minutes on CodeBuild - $1 per active CodePipeline (must run at least once a month to be considered active)
The only payment you'll encounter from Day One is an S3 storage fee for emails and CodePipeline artifacts. ```
So you can have a stack that is actually doing something very useful that costs not even a $1 a month.
It is possible to pay $0 to AWS, but you need to first understand AWS to be able to do it, another trivial example of a tiny project that is useful and cost $0 to run: https://github.com/0x4447/0x4447_product_secure301
The last point would be: don't listen to the marketing material - they are there to sell you AWS, marketing never cares about reality.
I also recommend this website https://awsvideocatalog.com - pick a service and watch all the keynotes AWS has on that service, if you'd spend 1h a day, in 6 months you'll know more about AWS then anyone else complaining here.
Anyway, life gose on, and it was overall a good chat :)
There's a big difference between owning up to your mistakes and taking on a bunch of unnecessary risk in situations where you know it's more likely you're going to make mistakes. IE: Learning, testing, etc..
In fact taking on unnecessary, possibly unlimited financial risk in a situation like that is a mistake. Want to own that one?
I don’t get the point that AWS docs are free. I mean, they are free, but the services are not. Do you pay for a visit in a car dealership?
Real insurance policies have maximum coverage limits, so, while it reduces your liability by a capped amount, it doesn’t, strictly speaking, limit it; you still face unbounded potential liability beyond your insurance coverage.
on edit: this is not to say that I believe an insurance for this kind of thing would really work (although maybe that is because I know nothing about the insurance business) just that I don't think the problem with the insurance would be that you could always end up getting billed more than you were insured for.
WHAT?
https://www.ibm.com/it-infrastructure/z/education/master-the...
https://sdl-hercules-390.github.io/html/
https://mainframe.informatik.uni-leipzig.de/
>There's a difference between "unable" and "unwilling".
True words, look at the Links.
Item 2 is documentation. This is good! But without access to the system to try it out, how self-teachable is it?
Item 3 is the one counter point. With a third party emulator though, how comprehensive/accurate is it? Would you hire someone to work on a mainframe who'd never used a real system before?
Compare that to e.g. Java. Sure there are plenty of college courses, some even Oracle affiliated, but I can also just download a Java compiler and start plugging away. .NET is there with Visual Studio Express. Oracle cloud, for everything else wrong with Oracle, has a free tier with no time limit. Microcontrollers for embedded development can be had for €20. You can install Linux or a Linux VM on your own computer. Even stuff that falls into the more enterprisey camp like MS SQL or Oracle DB has free personal licenses.
Attending a course on your Own.
>Item 2
No it's a MVS distribution (a full blown Operating System) with all the compilers 99% the same as a "modern" z/os
>Item 3
That's the S/390 emulator.
>Would you hire someone to work on a mainframe who'd never used a real system before?
Most Mainframe-Programmer where never near the Hardware anyway...so YES i would do that absolutely (you don't do Hardware near programming in Mainframes, an emulator is perfectly fine), same as a Dev who programs in a VirtualBox, no difference.
If you want to learn how to use it, and MUCH more:
https://www.youtube.com/channel/UCR1ajTWGiUtiAv8X-hpBY7w
>Compare that to e.g. Java.
You can do exactly the same with MVS, all the compilers (and more as z/os brings with) are in it.
Not a lawyer, etc, and don't even know what jurisdiction you are in or what providers you use, so this is just a general caution that your strategy may not generalize to other companies.
This is a hard no from me.
How is AWS going to even know what to shutdown/remove? What if it's storage causing my bill to overextend?
Yeah, not just no, but hell no.
My business is looking at cloud compute at the moment and we have absolutely no idea how to do it safely. In fact sagemaker is exactly a product we have looked at and discounted since we cannot be sure we can do it safely without getting unexpected megacharges.
We had an incident recently where a cellular router was set up to send a text message on a GPIO changing. The old firmware didn’t have pull-ups on the input. We told our end customer to update the firmware. They didn’t and also didn’t hook up the input and left it floating. We got a £20,000 bill for text messages as a result and a soured relationship with the customer.
It would be nice if cloud providers were better at surfacing potentially company killing things via their dashboards and nag emails.
For example, their CDN has no "TB per month limit". If your site gets popular and is 100% cacheable, you could be getting a million dollar bill.
I hear similar horror stories about the various distributed databases in several clouds. They're all crazy expensive.
AWS offers an AWS educate account already - go here to learn more
https://aws.amazon.com/education/awseducate/aws-educate-faqs...
If you are a startup go through AWS activate - 100,000 to play with.
GCP etc al do not offer this either. This is not done crime- AWS pioneered this free tier model. I used Dell and you paid for everything there.
As the article points out: Every other cloud does this! They all have non-production subscription types with hard spending limits.
There's a difference between "unable" and "unwilling".
The people that refuse to understand the difference are the same people that don't understand that "unsupported" isn't synonymous with "cannot be made to function".
Don't be that person.
If you have a large account and you're in regular contact with an AWS sales representative, pressure them into making this happen. Even if you work for Megacorp with a $$$ budget, keep in mind that your future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.
Don't walk down the same path as IBM's mainframes, where no student anywhere ever has been able to learn on their own, making it a dead-end for corporations who pay billions for this platform. You, or your company will eventually pay this price if AWS keeps this IBM-like behaviour up.
Think of the big picture, not just your own immediate personal situation.
Apply this pressure yourself, because the students you want to hire next year can't.