Whilst this might sound funny, we were surprised to see it as a common use-cases with users putting https://github.com/infracost/infracost in their CI/CD pipelines to act as safety net. Currently it only works for Terraform users, but we plan to add other infra-as-code tools in the future. We're also discussing how we can do this for people who don't use infra-as-code in https://github.com/infracost/infracost/issues/840 but it's not clear what the workflow could look like for them. Perhaps having separate AWS accounts with a budget alert that emails you to run https://github.com/rebuy-de/aws-nuke is a work-around just now.
(I'm co-founder of Infracost)
You absolutely must, MUST, MUST be using separate AWS accounts for separate purposes. You can have as many as you’d like and roll up the billing into one actual paying account.
This is a win for accountability (roll up dev and easily see the split out for separate environments), but more importantly for security as it limits the blast radius for any one environment. Combined with per-account budget alerts it’s a win across the board.
Does it make sense for one team to have 10+ AWS accounts per service because 'security'? How about if each team out of 1000s in your company has 10 AWS accounts per service?
We run our service in 3 geographic regions and have a separate AWS account for each region and stage despite each account supporting resources in multiple regions. Considering that we have 4~ services that is roughly 40 AWS accounts for just one team with less than 10 people.
What I'm describing above is the 'best practice' way to manage AWS accounts at scale. It is insane and saying 'security' does not magically make this reasonable.
Then I learned because they’re saving it all browser-side I had to rebuild the whole menu whenever I first used a new browser or computer? Whaaaat? Of all people, AWS console users have to be highly likely to be using multiple devices/browsers. Having to recreate your own prefs at each new environment is nuts.
Plus you have to look up the account id in order to set it up initially.
While security and UX are oftentimes in tension, in this case they don't have to be. It would not be that hard to be signed into multiple accounts and allow you to switch seamlessly between them (allow the tagging of each account, such that you can say, effectively, "show me dev us-east-1" vs "show me us-east-1" vs "show me dev", slicing and dicing between accounts that way). At that point, separating infra across accounts becomes semantically meaningful, and you can slice/dice in whatever way seems best (so you could have a full account for a single service, sure. Or an environment. Or a region. Or a combination of those, only service-Foo in us-east-1 for dev. Whatever level of granularity you want; trading off instead between the security of isolation with the convenience of colocation, which should be the actual UX cost; infra in the us-east-1 account has a harder time communicating with the infra in the us-west-1 account).
Users login to the Build Infra account and then Assume Role into the others - There's a list of magic links that does the assume role. There's also a list that is added to ~/.aws/config that does the equivalent: They configure one IAM key, and the rest are assumed automatically by the CLI or client libraries (Requires relatively recent client libraries; Java only started supporting this within the last year or two)
You can set budgets by project and easily allocate costs and address accountability issues across teams or products.
The value depends on how you operate.
I can see how starting with a pattern of "account per X" would create intuitive boundaries. When you say "per service" what kind of service do you mean? Business related web service API? AWS product? Other? Interested in what boundary line made sense for you given the large number of accounts you say you're happy with using.
Really soured me on the setup, tbh.
If someone acquires credentials, they are usually multi use and long term. And it can go unnoticed if an ec2 instance is span up running crypto mining on your dime, only for you to notice at the end of the day that your estimated bill has shot through the roof
A couple of fun billing surprises I've seen.
1. A bug in a system that uploaded quite a lot of data to Amazon S3 caused it to hit the S3 API to the tune of about $10K/day. Because AWS billing is usually 2-3 days lagged, it took us 3 days to notice. We fixed it right away once we found it. Goodby to that $30K.
2. An engineer did an Athena query that happened to walk many TB of data. And they unknowingly did it in us-west-2, but the data was in us-east-1. So that resulted in a cross region transfer to the tune of $10K for that single query.
Then last month I got an email saying "Hey, those quotas you were setting using the API documented to set quotas, those were actually not being enforced the whole time because of undocumented issues with our systems." So basically you can't rely on the documented behavior of these systems, there's no good way to test whether your code is correct or whether your limits will work without actually exceeding your budget for real, and the whole thing is a clusterfuck. When you get a surprise bill you just have to throw yourself at the mercy of whichever first line billing support rep is randomly assigned to your case.
Limiting your bill to something less than "potentially infinite" is just a basic fundamental feature that shouldn't require rolling your own bill-monitoring service relying on poorly documented and malfunctioning APIs with no provision for testing. There's no excuse strong enough to explain why the cloud providers can't do something reasonable here.
The "tiny bit of lag" between usage and billing calculation explodes when there's a lot of usage - in my case, a broken job tried resubmitting itself continuously, and the lag increased to 8 hours and $5000 just when I needed the alert the most. My team's response time was 5 minutes... After the 8 hour GCP lag.
Very similar to this guy's story: https://blog.tomilkieway.com/72k-1/
I had to go back and forth with them on email for weeks, and ultimately threaten them with a draft blog post with a lot of graphs and screenshots of their recommendations for them to cancel the bill.
I’d love it if GCP’s official method were to disable billing if your bill went over a limit.
Sadly, I suspect it would just disable systems instead.
Is this like asking the phone company "When I reach my plan limits, stop charging me money but let me keep making calls?"
I think GP was incorrect in the mechanism there.
I quoted GP verbatim, they said disable billing rather than disable service or delivery or system.
I use Vultr or Digitalocean if I need a server somewhere because at least it's just a pre-set cost.
I'm in love with DigitalOcean because you know the price you'd pay each month. If it's just 5 USD or 5000 USD, it's what you expected, nothing more.
I believe the rest of the clan (Linode, Vultr, etc.) give you the same certainty.
To be fair, AWS Lightsail should be an option too. Lightsail machines come with a fairly competitive amount of bandwidth.
That's exactly how it should work. It would even be useful if I could designate my development / testing account as unimportant by default so everything can be nuked to limit spending.
I think it's the kind of thing that will only be solved by regulation. The government needs to institute the concept of capped overages for cloud providers where if I set my budget to $100 / month they aren't allowed to send me a 100x bill for $10k.
Here's the 9 year old request for the same thing on Azure.
https://feedback.azure.com/forums/170030-signup-and-billing/...
I politely asked @AzureSupport (on Twitter) if they could have someone provide an update, but they didn't deliver on their promise to follow up :-(
The possibility that someone flood the server even for static resources causing bandwidth spiked Bill is scary.
Genuinely curious, is this just a side-effect of the cloud craze or did DDoS attacks become so powerful that old-school approaches of appropriately-sized bare-metal infrastructure with finite but unmetered bandwidth are no longer viable?
The way I see it, you can provision enough unmetered bandwidth to cover your typical load + a safety margin at a flat rate per month, and worst case scenario if the attack is big enough you merely get downtime (allowing you to re-evaluate the situation and decide whether to throw more bandwidth at the problem or purchase attack mitigation services) instead of an infinite bill?
My current ISP gives me 1Gbps unmetered. Worst case scenario the connection is saturated but at no point the ISP will come to me and ask for extra money.
The practical problem today is that cloud now has so much mindshare, justified or otherwise, that the ecosystem around private hosting is diminished. Finding good people with the required admin skills, good sources of equipment, even good software to run local versions of automation we take for granted in the cloud, can be harder than it used to be.
I won't be surprised if in a few years some huge tech firm we all thought had faded into obscurity enjoys a new lease of life by offering a set of locally hosted equivalents to popular cloud services that are also easy to administer and scale but come with a lot more predictability because they run on the customer's own infrastructure.
If you go up a few levels you can find more interesting things as well...
Someone wrote an good tutorial on using it here: https://codeseekah.com/2012/03/19/cross-server-deployment-wi...
I get the impression that some enterprise vendors don’t offer pay as you go solutions because it would put their sales staff out of work, and because they wouldn’t be able to use a “how much can you afford?” pricing model.
The limited protections available against this threat from the big cloud providers have to be seen as a warning sign. It's only a matter of time before any small business using these services for hosting can be subject to sudden shakedowns by criminals. "Nice business-critical infrastructure you have there, be a shame if anything were to happen to it." Some of the providers do offer a DoS mitigation service, but the cost for the higher levels can start to look like a shakedown itself.
This likely will just alert you somewhat quickly after something has spiked and been running for a number of hours/day, most likely.
My guess is that billing lags enough that they can't stick to a price cap, which means that they either have to guarantee the price cap and swallow the difference, which could be exploited by malicious users to get free compute, or they have to say that there's a delay on it which makes the cap fairly useless.
Some of these services are billed by such small increments I can't even imagine how complex billing for them is in practice. I'd be surprised if bills are eventually consistent within 24 hours.
I wouldn't be surprised if we see an announcement like billing being guaranteed after 1 hour at some point in the not too distant future, but I'd be surprised if we see realtime caps.
Trust me, they hear you.
Or maybe it is a costly implementation that would not bring any profits.
The strange thing is that the lack of this feature seems too incur a cost as it causes more calls to customer support. So, maybe it's that implement this feature will reduce profit more that it will reduce cost.
They nerfed the $100 of AWS credits for Alexa developers with zero notice this month, which caused me to incur overages this and last month.
I've gotten last month's bill waived, but still received a passive-aggressive email with bad English by a Territory Account Sales person from my region about how my account could be suspended, if I didn't reply to the email within the day. I'm not sure I would trust said person to handle my accounts, even if I was on a corporate budget.
I'm still in the process of moving most of my workload away from AWS.
All these stories of providers giving "good will" credit for these massive charges really concerns me when you look at how other parts of these companies ignore their customers or only reply with scripted responses.
At their revenue, don't care about 5K charge, they can send to collections / sell to 3rd party collections agencies.
They do care about keeping you happy as a customer since your employers will be swayed by their employees.
So the former is much more likely to succeed, the latter will just make you look like a scammer.
At larger sums - they will do much more rigorous checks to avoid issues.
I understand the role and the necessity for "the cloud", but it's a re-invention of the role of the mainframe. I hate seeing one of the most notable aspects of the microcomputer era go away which the ability of a motivated individual to gain computer skills using an individual's resources.
A publicly accessible mainframe, where anyone anywhere in the world can script the provision of machines and other resources with little more than terminal and a text editor.
That would have been utopian science fiction in the heyday of the mainframe.
[1] https://news.ycombinator.com/item?id=21694835
[2] https://wordpress.org/support/topic/amazon-cloudfront-invali...
Billing can be 24 hours delayed.
This inspired us to add billing limits to our SaaS product so that users don't have be in scary situations with bill run offs: https://mediamachine.io/blog/protect-your-customers-with-bil...
AWS, anecdotally, has removed 5k++ mistakes I’ve made with little question.
(One example they forgave due to my carelessness: ECS and Fargate service with logging to CloudWatch but with verbose logging on. The bill was 8k that month for just CloudWatch usage)
AWS is the one major tech company where I've never had any issue getting in touch with a real human who has been empowered to actually fix my issues.
The only thing that's been required from us was to show them we were taking reasonable steps to prevent it happening again.
AWS's unknowable policy for the cost of errors represents a huge risk for individuals and small businesses. It puts a lot of people off.
> AWS's unknowable policy for the cost of errors represents a huge risk for individuals and small businesses.
Assume you're paying for your own mistakes, and be thankful when you don't have to? That's a pretty easy policy.