Why? Because they serve just about every use case and scenario. I've myself spun up $5k/day resources on accounts that did four magnitudes less / day before that. There are some limits by default which you can get raised, but they're not going to prevent a $50k bill - at best, they're here to prevent a $500k one.
Anyway I agree they could make it clearer that you have to do all this crap yourself but it makes for a poor sales pitch.
Be a little patient, but don't try to superescalate by going through emails and what not. Just file a business ticket. If this needs escalation, they will do so.
This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms.
Your other option is to start a Cloudtrail and alarm on foreign IPs that are logging in, new IAM users and keys being created and changes to any alarms you have in place to check for this stuff. It won't necessarily stop it, but you'll be able to react a lot faster.
But what would be best practices for billing alarms? I have used them in the past when I used a bit more of AWS, but I don't think I ever got one (which is good).
But it happens to me that I miss emails that I would have liked to read in time for weeks. That could happen with a billing alarm, too.
Maybe you could forward it to SNS with SMS delivery. But as a matter of fact SMS is one of the few services (if not the only one) with a spending limit. If that is reached you silently won't get SMSes anymore, I have experienced that.
I'm not that worried though.
It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t seem to have any infrastructure monitoring?
I’d honestly hire some people that have experience in this area, because your “dev team” sounds clueless.
So yes, it was bad form for OP to not establish billing alerts, but you can't in any way say that AWS is transparent about most aspects of their service tiers.
AWS can certainly do a better job in telling me how to optimise my AWS hardware and expenses
In OPs case, if you search in Google, you will find this AWS hacking happen a lot with many users including me and AWS support was extremely kind to me to give additional credits to offset that expense.
It would be trivial for AWS to force you to setup spending limits when setting up an account, one for an alert, one for a hard lock.
AWS billing is terrible too, even now I'm getting charged a few dollars on my personal account for who knows what, I've cancelled everything I can find. It's like whack-a-mole everytime you spin something up.
In the end that's basically theft by AWS but you can't make too much noise or your account gets cancelled.
Anyway I was talking more about cloudwatch monitoring, including tracing all API calls. The billing is secondary: someone has been in his account and might have exfiltrated everything.
I myself have never created an instance, I set up the account then gave access to the devs. I only log into the account to provide new access to devs that need it and none of them are full time.
Ultimately the responsibility lies with me, but I would disagree that my dev team is clueless. Rather they're working on development, not watching what the servers are doing on a daily basis so I think that's a bit unfair.
I assume you revoke access later, but I doubt you audit anything that they may have done (like create keys that outlive their access) in the account or that any of it is version controlled or traceable.
And you’re surprised you’re in this situation?
And fair enough, you pay your contractors to do a specific job. None of them are going to point out that the way you’re managing your infrastructure is pretty slow and inefficient, or that perhaps there’s a better way to do any of what you’re doing on AWS that is cheaper, faster, more secure and that might give you a far quicker iteration time with the added advantage that you won’t fall apart with a surprise bill like this. They, after all, are working on “development”.
CloudWatch is standard AWS, or what do you mean?
Amazon gives you all the tools to monitor your usage and management of the cloud. You did not use those tools, had sloppy security and now it is Amazon's fault?
Take it as an expensive lesson learned. And get someone who knows what they are doing in AWS to administer it. Maybe your company was pennywise pound foolish by not having someone who knew AWS?
I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking the first went through.
Once access was made, we were completely unaware of their existence until I saw the charges and asked our devs about it. They said they didn't have any knowledge or access to these new instances.
I appreciate the advice, will upgrade the support and try again.