I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
My credit cards have credit limits, so it makes sense that a variable cost service should easily be able to support a spending limit too.
You do do stress testing ahead of peak season, right?
That’s an insane amount of both money and stress. You’re at Amazon’s mercy if they will or will not refund it. And while this is in process you’re wondering if your entire financial future is ruined.
If you go over your budget with AWS, what should AWS do automatically? Delete your objects from S3? Terminate your databases and EC2 instances? Besides, billing data collection doesn’t happen anywhere near realtime, consider it a fire hose of streaming data that is captured asynchronously.
Why not simply take the service offline once it reaches the free tier limit??
The reason why is that AWS is greedy, and would rather force you to become a paid customer…
If I know my service should never use more than 1TB total I'd like to be able to set a limit at (say) 2TB total with warnings at 0.6TB & 1TB, thus limiting spend to $46/month on storage. Sure, my service will fail if I hit the limit, but if it's using double the storage I expect it to use something went wrong & I want to require manual action to resolve it instead of allowing it to leak storage unbounded.
This is not a particularly difficult problem to make significant improvements on. There are some edge cases (there always are) but even if spending limits were only implemented for non-storage services it'd still be better for customers than the status quo.
The opposite problem when you do set low limits by default is that you constantly have to submit tickets to AWS to ask for service limit increases.
How is AWS suppose to know whether you want to immediately scale or not?
And before July of this year, there was no such thing as a “free tier AWS account”. There were services that allowed certain amount free.
Ask? This is not some impossible problem.
Yes, there is a UX challenge to be solved.
But also, doing so is well within the capabilities of a company like Amazon.
They simply have no incentive to help out since there is less money to be made by making it easier to spend less money. And, purely capitalistically, if you have to pick between a potential bug or misconfiguration that causes extra spending you can walk back with customer support, and a bug or misconfiguration that results in extra downtime for your 7+ figure customers, you pick the latter.
Their choice makes sense for their bottom line.
It's still bad UX for many users.
But as of July 15th of this year, there is actually a “free tier” that won’t let you spend over $200.
Before there were services with a free tier.
For a postpaid service with usage-based billing, there are no separate "free" and "paid" plans (= what you're clearly thinking of when you're saying "tiers" here.)
The "free tier" of these services, is a set of per-usage-SKU monthly usage credit bonuses, that are set up in such a way that if you are using reasonable "just testing" amounts of resources, your bill for the month will be credited down to $0.
And yes, this does mean that even when you're paying for some AWS services, you're still benefitting from the "free tier" for any service whose usage isn't exceeding those free-tier limits. That's why it's a [per-SKU usage] tier, rather than a "plan."
If you're familiar with electricity providers telling you that you're about to hit a "step-up rate" for your electricity usage for the month — that's exactly the same type of usage tier system. Except theirs goes [cheap usage] -> [expensive usage], whereas IaaS providers' tiers go [free usage] -> [costed usage].
> Amazon should halt the application when it exceeds quota.
There is no easy way to do this in a distributed system (which is why IaaS services don't even try; and why their billing dashboards are always these weird detached things that surface billing only in monthly statements and coarse-grained charts, with no visibility into the raw usage numbers.)
There's a lot of inherent complexity of converting "usage" into "billable usage." It involves not just muxing usage credit-spend together, but also classifying spend from each system into a SKU [where the appropriate bucket for the same usage can change over time]; and then a lot of lookups into various control-plane systems to figure out whether any bounded or continuous discounts and credits should be applied to each SKU.
And that means that this conversion process can't happen in the services themselves. It needs to be a separate process pushed out to some specific billing system.
Usually, this means that the services that generate billable usage are just asynchronously pushing out "usage-credit spend events" into something like a log or message queue; and then a billing system is, asynchronously, sucking these up and crunching through them to emit/checkpoint "SKU billing events" against an invoice object tied to a billing account.
Due to all of the extra steps involved in this pipeline, the cumulative usage that an IaaS knows about for a given billing account (i.e. can fire a webhook when one of those billing events hits an MQ topic) might be something like 5 minutes out-of-date of the actual incoming usage-credit-spend.
Which means that, by the time any "trigger" to shut down your application because it exceeded a "quota" went through, your application would have already spent 5 minutes more of credits.
And again, for a large, heavily-loaded application — the kind these services are designed around — that extra five minutes of usage could correspond to millions of dollars of extra spend.
Which is, obviously, unacceptable from a customer perspective. No customer would accept a "quota system" that says you're in a free plan, yet charges you, because you accrued an extra 5 minutes of usage beyond the free plan's limits before the quota could "kick in."
But nor would the IaaS itself just be willing to eat that bill for the actual underlying costs of serving that extra 5 minutes of traffic, because that traffic could very well have an underlying cost of "millions of dollars."
So instead they just say "no, we won't implement a data-plane billable-usage-quota feature; if you want it, you can either implement it yourself [since your L7 app can observe its usage 'live' much better than our infra can] or, more idiomatically to our infra, you can ensure that any development project is configured with appropriate sandboxing + other protections to never get into a situation where any resource could exceed its the free-tier-credited usage in the first place."
Oracle Cloud has a 30-day free trial; and that free trial seems to have had some dedicated effort put into a whole divergent billing-infra path for it.
Under Oracle Cloud's free trial, you get a certain amount of spend ($300 in credits); and then, when your trial either expires (30 days) or you run that credit pool down to zero, your account is shut off.
Oracle do eat any marginal costs from your spend taking your credits "below zero" before they shut the account off, because your account was never billing to you anyway; it was billing to Oracle's marketing department as a lead-gen expense.
In other words, unlike Oracle Cloud's steady-state IaaS offering, their free-trial IaaS offering is actually a prepaid (but usage-billed) paradigm — with Oracle being the ones doing the pre-payment.
This works much like an oldschool prepaid phone plan, where you pay in every month to be given a certain number of [expiring/non-"rollover"] minutes/texts/MB of data; and then you get an itemized invoice at the end of the month for how close you came to "using up" each resource that month. And you very well can use up a resource's monthly paid allocation before the end of the month — e.g. "running out of texts" and being unable to send more, rather than those converting into something billed to you. (In a prepaid context, that "converting into being billed" is called "flex" or "pay-as-you-go" [PAYG] billing, and is usually some extra option you would have to enable, if offered at all.)
At scale, prepaid usage-billed systems are also asynchronous; to continue the telecom analogy, most phone-service providers won't re-aggregate your prepaid calling minutes to notice you've run out, until you hang up your current call. Only rarely do they have infra where the billing system can ping the telecom switches' control planes to say "hey, this guy just went over, hang up the call" — and when they do, they only do such checks on a 5-minute/30-minute interval, probably as a scheduled batch query.
But, yes, prepaid systems almost always do just eat any overage generated by this detection gap. This is usually safe, because prepaid systems are almost never elastic to the point that you could accrue nontrivial expenses during that short accounting gap.
When a system is that elastic, a systems architect responds by saying "this should be a postpaid system."
Which means that Oracle Cloud's free trial — insofar as it allows you to make use of truly-elastic resources with per-credit upstream basis costs, like FaaS compute — is probably vulnerable/exploitable. Oracle may sometimes be eating some hefty bills, where people on a free trial have wired their FaaS into a proxy fronting some already-highly-popular service.
This is mostly fine, if you have Oracle's treasury, because you'll still be doing KYC in advance of giving out these trials, so you'll only be letting any given individual do one trial.
But this does put Oracle in the territory of "having to think about people who buy burner identities on the black market [usually for ~$1] to sign up for services using them" + "having to think about people who sign up for their free trial and then sell that free-trial account's credentials on the black market [again, usually for ~$1]."
I haven't checked myself, but I would guess that like any other provider who sees this type of attack (e.g. Hetzner), Oracle Cloud likely has hardened registration flows that reject identities + cards from certain parts of the world; traffic fingerprinting heuristics that immediately shut down free trials if they start up a DDoS attack or the like; etc.
Which is something the other clouds get to skip thinking about entirely, by not having a true "free trial" with a prepaid model, and instead just offering e.g. a one-time $300 sign-up-bonus account credit.
---
But remember, we're only talking about the "free trial" here — something you only get access to for the first 30 days.
Oracle's free tier — the thing you have after the first 30 days — is no different than the one every other IaaS offers. It needs a billing account populated by your credit card; there's infrastructure to allow you to automate control-plane actions in response to billing thresholds being hit, but no offering that will wire anything up for you; etc.
In Oracle Cloud's free tier, you can set budget limits that will prevent new costed resources from being leased while your account is over that limit in a given month (which is certainly nice) — but those budget limits don't affect ongoing usage-based-billing of a resource. Your FaaS endpoints will continue to accrue vCPU-seconds of billed usage, until you — or some automation you wrote — shuts them off.
I'd rather they didn't have the option, I'd rather not stake my financial future on the good graces of Amazon.
Particularly the other side of Amazon.
On serverless, I can enter numbers in a calculator and guess that running my little toy demo app on AWS will cost between $1 and $100. Getting hit with a huge $1000 bill and a refusal to refund the charges (and revocation of my Prime account and a lifetime ban from AWS and cancellation of any other services I might otherwise run there) would be totally possible, but I have zero control over that. Expecting to go on social media begging for a refund is not a plan, it's evidence of a broken system - kinda like those "heartwarming" posts about poor people starting a GoFundMe so their child can afford cancer treatment. No, that's awful, can we just be sensible instead?
If a server would have cost me $20 at a VPS provider to keep a machine online 24/7 that was at 1% utilization most of the time and was terribly laggy or crashed when it went viral, that's what $20 buys you.
But, you say, analysis of acttual traffic says that serverless would only cost me $10 including scaling for the spike, in which case that's a fantastic deal. Half price! Or maybe it would be $100, 5x the price. I have no way of knowing in advance.
It's just not worth the risk.
ofc. When things go viral they say "yes". But i would really love to get some number how many students and hobbiests got a 1k-2k bill and just paid it for the problem to go away.
Amazon is a publicly traded company. If they wave fees every time something goes wrong, investors would tell them something.
Have you read one even anecdotal case where AWS didn’t immediately give a refund to a student who made a mistake just by them asking?
I haven’t even seen reports on subreddits
By the way, there is nothing stopping AWS from reversing their trend of issuing refunds for big mistakes. "It hasn't happened in the past" isn't a great way to argue "it won't happen in the future."
They give you the tools. It’s up to you to use them. If that’s too difficult, use the AWS LightSail services where you are charged a fixed price and you don’t have to worry about overages or the new free tier
Because despite what everyone here is saying, before July of this year, there was no such thing as a free tier of AWS, there was a free tier of some of their services
If it is so easy to set up these automations, Amazon can easily set up this automation for you. Ask yourself why they don't.
Also a vital lesson from the big tech companies that sell a wide variety of services: don't get your cloud hosting from a company that you also use other services from.
I had to disable photo syncing because Google photos eats up my Gmail space. Having Amazon's cloud billing fuckup threaten your TV access is another level.
We clearly need to keep the option open to burn those bridges.
In any case, if I ever host anything, I'm going to host it from my home.
At minimum they should provide hard billing caps.
intentionally allowing huge billing by default is scummy, period.
All else being equal, would you rather choose the platform where a junior dev can accidentally incur a $1M bill (which would already bankrupt early startups), or the platform where that same junior dev get a "usage limits exceeded - click here to upgrade" email?
I also wouldn’t give a junior dev access to production databases.
Also from working with AWS from both the inside (Professional Services) and the outside at a third party consulting companies, I know how aggressively AWS is about keeping startups and they would never risk losing the continuing revenue of a company like that.
If a junior dev has the access to do that, then there is a big failure (probably more than one) by someone who isn't a junior dev after choosing AWS that was necessary to enable that.
Taxes are different because they never exceed the amount the person paying the taxes receives.
That would solve this issue, but needs to be an explicit opt-in, because a lot of users don't want that.
Like they use to say about IBM…
“No one ever got fired for choosing AWS (or Azure)”
If you woke up to the auto-withdrew $100k from your bank account and now you need to get it back that's another story.
They do this and make it easy to get a refund because for every demo account that does it some bigger account accidentally gets billed 10K and they have to pay for it. They have skin in the game and cannot risk their account being down for any time period.
As I asked before, if what is causing overages is not web requests but storage should they just delete everything?
Counter real world example. I was doing some consult work for a place that had a 9k unexpected charge on AWS. They asked about it and I explained about how they could dispute it. They said ugh never-mind and just paid it. FYI it was a charity which I've since learned its common for charities to be wasteful like this with money since they are understaffed and OPM(Other peoples money)
While I have no love for AWS as an employer, I can tell you that everyone up and down the chain makes damn sure that customers don’t waste money on AWS. We were always incentivized to find the lowest cost option that met their needs. AWS definitely doesn’t want customers to have a bad experience or to get the reputation that it’s hard to get a refund when you make a mistake.
Of course AWS wanted more workloads on their system.
Semi related, AWS will gladly throw credits at customers to pay for the cost of migrations and for both inside and outside professional services firms to do proof of concepts.
For storage specifically, in that circumstance, if you weren't hellbent on claiming otherwise: it's easy to figure out what to do. For storage: block writes and reach out to the customer. Also, people are extremely unlikely to accidentally upload eg 250tb which is how you'd get to, say, $200/day. Whereas similar bills are extremely easy to accidentally create with other services.
It's totally reasonable to want spend limits firmer than AWS' discretion, which they can revoke at any point in time for any reason.
Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money.
But I don't want to be forced into bankruptcy because my five visitor a week demo project suddenly becomes the target of a DDOS for no reason at all and the hosting company decides this isn't a "usually" so please send the wire transfer.
https://docs.aws.amazon.com/cost-management/latest/userguide...
What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500.
I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 90%. It’s great, but it’s not a limit. I can still get screwed if somehow, my projects become targets, and I don’t see the emails immediately or aren’t able to act on them immediately.
It blows my mind there’s no way I can just say, “there’s no conceivable outcome where I would want to spend more than $10 or more than $100 or whatever so please just cut me off as soon as I get anywhere close to that.”
The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects, and thus exposing yourself to ruinous liability.
I suppose you could bake the limits into each service at deploy time, but that's still a lot of code to write to provide a good experience to a customer who is trying to not pay you money.
Not saying this is a good thing, but this feels about right to me.
And frankly any pay-as-you-go scheme should be regulated to have maximum spending limit setting. Not only in IT.
If you did this to a vps the instance might be unavailable until the traffic slows down but up after.
Trust me you would rather they freeze your account.
Is that not a serious request? I play around in the same big-boy cloud as some SaaS company, but I'm on the free tier and I explicitly do not want it to scale up forever, and I explicitly do not want to destroy my credit or even think about having to call Amazon over a $100,000 bill because I set my shit up wrong or whatever. I want it to shut off my EC2 instance once it has used up whatever amount of resources is equal to $X.
Obviously any world with this feature would also feature customizable restrictions, options, decision trees, etc etc. I don't think anyone is or was suggesting that someone's SaaS app just gets turned off without their permission.
If you're worried about getting a big bill, and you don't care if it gets shut off when you're not using it, why don't you shut it down yourself?
AWS made the tradeoff to keep the lights on for customers and if there is a huge bill run up unintentionally and you contact them with it they refund it. I've never experienced them not doing this when I've run up five figure bills because of a misconfiguration I didn't understand. I don't think I've ever even heard of them not refunding someone who asked them for a refund in good faith.
There were no warnings or "don't do it again". They, very reasonably IMO, asked us to essentially explain how and why this happened and how we'd stop it happening again. They then provided some additional guidance and resources around those areas. In the one case where the charges were due to compromised credentials, they asked us to rotate all of our access keys before they issued the refund.
Completely anecdotal and slightly dated information, but that's been my experience.
If you try adding files that will result in a larger bill than your limits over the billing period you warn and refuse.
So simple.
You're right in that there's a few services that expose this complexity directly, the ones where you're paying for actual storage, but this is just complex, not impossible.
For one thing, storage costs are almost always static for the period, they don't scale to infinite in the same way.
https://medium.com/%40akshay.kannan.email/amazon-is-refusing...
But, I don’t think the idea of just stopping charging works. For example, I had some of their machine image thingies (AMI) on my account. They charged me less than a dollar a month, totally reasonable. The only reasonable interpretation of “emergency stop on all charges completely” would be to delete those images (as well as shutting down my $500 nodes). This would have been really annoying, I mean putting the images together took a couple hours.
And that’s just for me. With accounts that have multiple users—do you really delete all the disk images on a business’s account, because one of their employees used compute to hit their spend limit? No, I think cloud billing is just inherently complicated.
I disagree; a reasonable but customer-friendly interpretation would be to move these into a read-only "recycle bin" storage for e.g. a month, and only afterwards delete them if you don't provide additional budget.
https://docs.aws.amazon.com/accounts/latest/reference/manage...
I'm sorry but this is complete bullshit. they can set a default limit of 1 trillion dollars and give us the option to drop it to $5. there's a good reason they won't do it, but it's not this bullshit claim that's always bandied about.
What happens to your files when you refuse to pay the bill?
Just saying that there should be a limit is not an explanation.
(for my hobby projects, I'm happy for the limit to be $5 and delete everything when the limit is reached. that's what my backups are for.)
You don't stop CHARGING. You stop providing the service that is accumulating charges in excess of what limit I set. And you give some short period of time to settle the bill, modify the service, etc. You can keep charging me, but provide a way to stop the unlimited accrual of charges beyond limits I want to set.
> No, I think cloud billing is just inherently complicated.
You're making it more complicated than it needs to be.
> The only reasonable interpretation of “emergency stop on all charges completely” would be to delete those images.
It's by far certainly not the 'only reasonable interpretation'.
"Stop all charges" is a red herring. No one is asking for a stop on charges. They want an option to stop/limit/cap the stuff that causes the charges.
Might work. I do think that part of the appeal of these types of services is that you might briefly want to have a very high $/sec. But the idea makes sense, at least.
As someone else pointed out, some(?) services prevent unlimited autoscaling, but even without unlimited, you may still hit a much larger limit.
Being able to say 'if my bill goes above $400, shut off all compute resources' or something like that. Account is still on, and you have X days (3? 1? 14?) to re-enable services, pay the bill, or proceed as you wish.
Yes, you might still want some period of high $/sec, but nearly every horror story in this vein ends with an issue with the final bill. Whether I burn $300 in 5 minutes or 26 days, I want some assurance that the services that are contributing most to that - likely/often EC2 or lambda in the AWS world - will be paused to stop the bleeding.
If you could pipe "billing notification" SNS message to something that could simply shut off public network access to certain resources, perhaps that would suffice. I imagine there's enough internal plumbing there to facilitate that, but even then, that's just AWS - how other cloud providers might handle that would be different. Having it be a core feature would be useful.
I was on a team that had our github CI pipeline routinely shutdown multiple times over a few weeks because some rogue processes were eating up a lot of minutes. We may have typically used $50/$100 per month - suddenly it was $100 in a day. Then... $200. Github just stopped the ability to run, because the credits used were over the limits. They probably could run their business where they would have just moved to charging us hundreds per day, perhaps with an email to an admin, and then set the invoice at $4500 for the month. But they shut down functionality a bit after the credits were exhausted.
There are services that give you a free year and there are services that give you a free amount every month.
If you want AWS with training wheels, use AWS Lightsale
Compute and API access to storage is usually the thing that bites people with cloud costs.
I want an option that says if I go over $20 on my lambda costs for lambda X, shut it off. If I go over $10 on s3 reads, shut it off.
I missed a water bill payment years ago. They shut off the water. They didn't also come in and rip out all my plumbing and take every drop of water from the house.
Most (aws) services support limits which prevents unlimited autoscaling (and thus unlimited billing)
That _also_ runs into problems!
Take, for example, a nightly job that spins up a few giant instances to do some batch processing and shuts them down. Running an hour a night, over the course of the month that's going to accumulate ~$300 in charges. Great, we can set a $400/mo budget and have some wiggle room and all is well!
But how can AWS know that you're going to shut the instances down? Looking only at the rate charges are accumulating, the first night those instances start up you are on track to run up a $7,000 bill!
So do we set a $400/mo budget and then just kill the account so it stops accumulating charges when we hit $400, or do we set a $7,000/mo budget to account for the potential rate of accumulation and risk exceeding our budget by 2,000%?
It would be nice if this were in fact just overcomplicating things, but after much thought and many arguments on the internet I really can't see an easy "general" solution to this. The solution is heavily dependent on your specific workload and usage patterns, and the tooling is there to manage that if you want: Create billing alerts, and run code to adjust your usage in response to them.
That all said: I would fully support some sort of "developer sandbox" account that allowed a "kill the account" billing limit. I'd really prefer it had some sort of obvious limitation to avoid people accidentally using it for production workloads or dev workloads turning into production ones. Something like a hard limit that shuts the account down in 30 days, or limiting inbound connectivity to only via a VPN or something. That's purely self interest though--I don't want to see the article on the top of HN every few weeks about how "Amazon killed my startup" because someone set a billing limit and then all their customers' data was deleted.
The difference is that cloud providers charge you for the “at rest” configuration, doing nothing isn’t free.
Surely these billion and trillion dollar companies can something so basic.
Is that worth the support to refund the 10k and 100k charges? Maybe it is.
Yeah, I'm sure this is it. There is no way that feature is worth the investment when it only helps them sell to... broke individuals? (no offense. Most individuals are broke compared to AWS's target customer).
> There can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge. This is due to a delay between when an AWS resource is used and when that resource usage is billed. You might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you, and your actual costs or usage may continue to increase or decrease after you receive the notification.
Is this a perfect solution: no Is this still a solution: yes
And as others have also mentioned, the reports have a delay. In many cases it’s several hours. But worst case, your CURs (Cost usage reports) don’t really reflect reality for up to 24 hours after the fact.
This is a reason why I am not only clueless of anything related to cloud infrastructure unless it's stuff I am doing on the job, nor I am willing to build anything on these stacks.
And while I guess I have less than 10 products build with these techs, I am appeal by the overall reliability of the services.
Oh lastly, for Azure, in different European regions you can't instance resources, you need to go through your account representative who asks authorization from the US. So much for now having to deal with infrastructure pain. It's just a joke.
That's one positive side of Azure.
[1]: https://learn.microsoft.com/en-us/azure/cost-management-
You get a warning. There's no service cutoffs or hard limits on spending.
You can ring up tens of thousands+ overnight with AWS. The scale of potential damages is nowhere even close.
The broken water pipe should be covered by buildings insurance, but I can imagine it not being covered by some policies. Luckily a broken water pipe is likely not as expensive as not having e.g. third party liability protection if part of your roof falls off and hits someone.
For the cloud, I have the good will of the cloud provider and appealing to social media. Not the same thing.
Unlike cloud services, your electrical service has a literal circuit breaker. Got a regular three-phase 230V 25A hookup? You are limited to 17.25kW, no way around that. If that shithead neighbor tries to draw 50kW, the breaker will trip.
If it were the cloud, the power company would conveniently come by to upgrade your service instead. A residential home needing a dedicated 175MW high-voltage substation hookup? Sure, why not!
Water leaks, on the other hand, tend to be very noticeable. If a pipe bursts in the attic you'll end up with water literally dripping from the ceiling. It is very rare to end up with a water leak large enough to be expensive, yet small enough to go unnoticed. On the other hand, the cloud will happily let your usage skyrocket - without even bothering to send you an email.
There are plenty of compute service providers working with a fixed cap, a pre-pay system, or usage alerts. The fact that the big cloud providers don't is a deliberate choice: the goal is to make the user pay more than they wanted to.
As always, it just doesn’t make an awful lot of sense to compare physical and virtual worlds. As in leaving your front door unlocked in rural areas vs not securing your remote shell access.
We can't implement a basic cost limiter policy.
I think we all know why.
It's entirely possible to build cloud first solutions that scale better and are cheaper than your standard reliable colo solutions. But you've got to understand the tradeoffs and when to limit scaling otherwise things can run away from you. I still reach for "cloud first" tools when building my own projects because I know how to run them extremely cheaply without the risk of expenses blowing up because some random thing I've built lands on HN or the equivalent. Many hobby projects or even small businesses can leverage free tiers of cloud services almost indefinitely. But you've got to architect your solutions differently to leverage the advantages and avoid the weaknesses of the cloud. Actually understand the strengths and limitations of the various cloud "functions as a service" offerings and understand where your needs could be solved from those tools and how to work within those cost constraints. Repeatedly I see people trying to use the cloud as if it's just another colo or datacenter and build things in the same way they did before and only think about things in terms of virtual machines tend to have a more difficult time adopting the cloud and they end up spending far more than the companies who can tear down and spin up entire environments through IaC and leverage incremental pricing to your benefit.
There's no need to imply that, it's not illegal to criticise AWS. They do not want anybody to be able to set a limit on spend as that would probably hurt the business model.
They get a nice tax write-off.
It's couch-cushion change for them, but it adds up. They have whole armies of beancounters, dreaming this stuff up.
It's also the game behind those "coupons" you get, for outrageously-priced meds that aren't covered by insurance.
If they charge $1,000 for the medication, but give you a "special discount" for $90%, they get to write off $900.
Businesses are only taxed on actual revenue earned.
What you decide to charge—whether $100, $50, or even giving it away for free—is purely a business decision, not a tax one.
—
This is different from a nonprofit donation scenario though. For example, if your service normally costs $X but you choose to provide it for free (or at a discount) as a donation to a non-profit, you can typically write off the difference.
I’ve heard stories like this, many times, from businesses people.
They certainly believe in the pattern.
I don't want to go too far down the rabbit hole of hn speculation, but if another entity owes you 100k, and they go bankrupt, there absolutely are tax implications.
That is a lack of payment situation.
Revenue was still earned (and charged) … and since you never collected revenue then you don’t pay taxes.
The second one (the prescription one) may well be wrong. It’s total speculation, on my part.
The first one, though, I’m pretty sure is right.
I love the traditional debate tactic, where we find one part of an argument to be wrong, and use that, to say the other part is, too.
It’s no matter. I’m done here, anyway, and yield to you. The fox ain’t worth the chase.
Didn't the bootcamp told you to, at least, setup a budget alert?
I'm not trying to reduce AWS' responsibility here, but if a teaching program tells you to use AWS but doesn't teach you how to use it correctly, you should question both AWS and the program's methods.
They refunded you $100k with few questions asked, and you hate them for it?
I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them.
I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”
Terrible analogy.
They exposed him to 100K of liability without any way to avoid it (other than to avoid AWS entirely), and then happened to blink, in this case, with no guarantee that it would happen again. If you don't happen to have a few hundred thousand liquid, suddenly getting a bill for 100K might well be a life-ruiningly stressful event.
This is the problem right here. I moved from AWS and specifically Beanstalk because I don't want to be some "certified AWS goblin". I just wanted to host something sensibly.
Other hosting companies don't have this problem and while I cannot complain about AWS as a service, this can be improved if there would be the will to do so. I believe there are other incentives at work here and that isn't a service to the customer.
There were free services up to a certain usage limit in a month.
I mean, S3 also incurs ongoing charges, so if you're going to stop accruing charges you'd also be deleting your data that wasn't on ephemeral storage...
And potentially deleting all of your DNS zones (and recreating them will likely give you different nameservers so you'll need to wait for the registrar to update them once you're back)...
And...
its so easy to get billed a ridicules amount if money
Is it just me or is this just a cheap excuse to grab a payment method from unsuspecting free-tier users?
With that said, AWS is notoriously opaque in terms of "how much will I pay for this service" because they bill so many variable facets of things, and I've never really trusted the free tier myself unless I made sure it wasn't serving the public.
For you to put together this sort of argument with a straight face, you need to have little to no understanding of how the internet in general, services and web apps work. Good luck arguing your case with a judge.
In this situation if it were to actually go to court I’d expect the actual bill to get significantly reduced at the very least. But because it’s a completely bullshit charge based on bandwidth usage (which costs them nothing) it won’t go anywhere near that and will just get written off anyway.
I feel like this brand of sentiment is everywhere. Folks want things simple. We often figure out what we need to do to get by.
Over time we learn the reason for a handful of the options we initially defaulted through, find cause to use the options. Some intrepid explorers have enough broader context and interest to figure much more out but mostly we just set and forget, remembering only the sting of facing our own ignorance & begrudging the options.
This is why k8s and systemd are have such a loud anti-following.
By that logic, any technology that you can get certified in is too complicated?
Most systems are now distributed and presenting a holistic view of how it was designed to work can be useful to prevent simple mistakes.
Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
That is a common view in UX, yes. It's a bit of an extreme view, but it's a useful gut reaction
> Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
In the US roads are designed so that you need as close to no knowledge as possible. You need to know some basic rules like the side of the road you drive on or that red means stop, but there is literal text on common road signs so people don't have to learn road signs. And the driving license is a bit of a joke, especially compared to other Western countries
There is something to be said about interfaces that are more useful for power users and achieve that by being less intuitive for the uninitiated. But especially in enterprise software the more prevalent effect is that spending less time and money on UX directly translates into generating more revenue from training, courses, paid support and certification programs
In IT, I am inclined to agree with that. In real engineering, it's sometimes necessary, especially dangerous technology and technology that people trust with their life
Software runs on so many things we depend on IMO it also in many cases falls in the "dangerous technology" category.
Non-hobby OSes, non-hobby web browsers, device drivers, software that runs critical infrastructure, software that runs on network equipment, software that handles personal data, --IMHO it would not be unreasonable to require formal qualifications for developers working on any of those.
But infact, it is intended side effects. Things like Jaywalking or "no spitting" laws let police officers harass more people _At their whim_. And they're fullying designed that way but left as "unintended" for the broader public scrutiny.
So, just like, learn that "logic" is not some magic thing you can sprinkle on everything and find some super moral or ethic reality. You have to actually integrate the impact through multiple levels of interaction to see the real problem with "it's just logic bro" response you got here.
It is a fake degree.
> Do you want safeguards to be enabled by default, so you have to disable manually those you want to resign from?
OR
> Do you want safeguards to be disabled by default, so you have to enable manually those you want to be in place?
To then rebel against it and say "I lose seconds of my life reading this, I don't want to have to!" would be ridiculous.
Even if you rig up your own spending watchdog which polls the clouds billing APIs, you're still at the mercy of however long it takes for the cloud to reconcile your spending, which often takes hours or even days.
It’s basic stuff.
You forgot to mention Stanley Tools paid for the hospital bill.
If a tool is designed for experts, but you as the manufacturer or distributor know the tool is used by general populace, you know it's being misused every now and then, you know it harms the user AND YOU KNOW YOU BENEFIT FROM THIS HARM, AND YOU COULD EASILY AVOID IT - that sounds like something you could go to jail for.
I think if Amazon was a Polish company, it would be forced by UOKiK (Office of Competition and Consumer Protection) to send money to every client harmed this way. I actually got ~$150 this way once. I know in USA the law is much less protective, it surprises me Americans aren't much more careful as a result when it comes to e.g. reading the terms of service.
I call bullshit
> as in how am I going to pay it?
Really?
Amazon charged your card for $100,000 and your bank allowed it?
You're filthy rich by most people's standard, and you were able to pay it.
Amazon was operating in such a good faith that they ate the computational cost you spent. And you hate them for this to this day...
That would make you one of the most successful websites on the internet, or the target of a DDoS -- which was it? I assume you're not saying that "bots" would randomly hit a single, brand-new "hello world" site enough to generate that kind of bill.
Given the relative accessibility of stolen credit card info, isn't the CC-as-ID requirement easy for a criminal to bypass?
It's the easiest thing in the world - they just don't want to because they figured that they could use their scale to screw over their customers. And now you have the same guys who screwed everyone over with cloud compute wanting you to pay for AI by using their monopoly position to charge you economic rents. Because now things like edge compute is easy because everyone overspent on hard drives because of crypto. And so you have jerks who just move on to the next thing to use their power to abuse the market rather than build credibility because the market incentivizes building bubbles and bad behavior.
Smart evil people who tell others "no you're just too dumb to 'get it' (oh by the way give me more money before this market collapses)" are the absolute bane of the industry.
It's weird that you have people in here defending the practice as if it's a difficult thing to do. Taxi cabs somehow manage not to charge you thousands of dollars for places you don't drive to but you can't set up an if statement on a server? So you're saying Amazon is run by people that are dumber than a taxi cab company?
Ok, well you might have a point. And this is how Waymo was started. I may or may not be kidding.
AWS also provides training and education on how to use their services. If launching a "hello world" Elastic Beanstalk instance is so dangerous, why doesn't the tutorial require you to first provide proof that you are an AWS Certified Cloud Practitioner?
An idea I can stand behind. Or do you just let any "self-learner" take care of your banking Oracle or IBM DB2 database...?
It does on the surface, but what doesn't make sense is to register with a credit card and not read the terms very carefully: both for the cloud service and for the bank service.
In this aspect cash is so much better because you have only one contract to worry about...
Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.