I don't get time for that - I've got access to Cloudability but no one has ever showed me how to use it.
- run on demand instances (maybe mix in spot)
- run fargate instances (maybe mix in spot)
- reserve them.
Every company hates #3, but it's usually the best choice if you can't go spot - you can swap them within the family for size, use them for baseline, it's fine. You're incorporated right? So if you fold, it's just another debt on the pile, stop worrying. Go no upfront if you really want to avoid the cash outlay, or go spot if you can defend limited outages.
If you have a 1-year project that will have a 1-year extension, then get a 3-year reservation (with some amount upfront) that you can sell.
Your second best option is a Savings Plan.
- Engineers may want to switch sizes or go serverless.
- Long term vendor lock-in.
- Miss out on price cuts.
There’s a reason the vendors push them so hard. (No spot market on Azure either)
Surprisingly enough we have 3 people working on it right now.
Sadly they’ve found that it was extremely easy to save several thousands of dollars a day, and has been so for the past two years.
Terrible for who? You or the cloud provider? I'm a bit of a worrier (and not a perfect coder), so infinite scalability with utility billing combined with a mistake on my part could result in an epic bill. All I'll say is that I won't practice cloud development on my own credit card.
In my experience it takes azure support over a week to answer any question. Unless the question happens to be which person without a substantial update is taking over the case. You’ll get lots and lots of those answers.
Someone else's computer has been painful and very expensive for our organisation so far. Escalating costs aren't even the worst problem.
Wrote to support, explained my situation, and they removed the bill without any hassle.
Azure forced me onto invoice billing against my will - and every month without fail they don't automatically match my payment and I have to endure increasingly threatening emails that my account is pending termination while their support takes days to reply to my tickets about it, and when they do reply the advice is usually just 'wait a little longer'.
I've moved critical services off Azure because the risk of wrongful termination is just too high.
Check your partial / interrupted multi-part uploads. If its like amazon the uncommitted blobs won't show up in storage graphs but you'll be getting charged for them just sitting there.
AWS in particular is handsomely engineered so you can't answer this sort of question, as well as making people spend more without need.
- Per resource cost is turned off by default. Turning it on costs money. How much? ¯\_(ツ)_/¯
- Services like S3 will bill you per API call. How do you know who is calling your APIs? Using Cloud trail, which is turned off by default. Turning it on for this sort of analysis might cost a lot $$$ for a reasonably used bucket.
- CloudWatch will bill you per custom metric and adds up quickly. Why do I have so many custom metrics? Good luck finding out.
- Many costs are tied with stuff that's really hard to control e.g. egress traffic.
- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc...
- Many services are ha, multi az only. That makes AWS unnecessarily expensive for dev environments / non critical services.
The list goes on. I know a fair number of consulting companies making the money of their lives reducing AWS bills.
Not if you use shared VPCs.
And you can have business support only on one account... obviously you can go Enterprise and have the same support level on everything but pay a lot more $$$
It just always seems a bit salesie to me. Bring your network switching to the "cloud".. yadda yadda. Why? Everything is working great.
The problem is that mostly anywhere else in the world, you can get personell for much cheaper.
For highly volatile workloads, ephemeral workloads, anything where go forward capacity is fuzzy or unknown, cloud is awesome. For anything where your margins aren't robust, or your workload is known and consistent, you're better off going through the schlep of internal IT management (as long as your org isn't dysfunctional and how you operate this is a business enabler, not a hinderance, forcing shadow IT attempts). If you can't generate more revenue or margin, you've gotta look at cost savings, including the cloud premium.
The thing is, I've yet to see any startup even with VC money that has a product which has this need. Most startups nowadays factor away common crud, like booking a service or providing organisational tools. These high pressure, intense workloads are often incredibly unrealistic.
The scalable cloud is useful for scalable workloads (AKA how many times is you peak demand larger than the average?). If you want managed infrastructure, you are looking for renting a managed server or VPS.
Sounds like the more basic problem is that we keep end up working in companies where the "higher ups" are constantly shoving decisions down our throats.
Which, ultimately, is probably the root cause of (most of) the caginess, the yak shaving, the compunction to use cloud databases and cloud-everything, and the silly support tickets that stay open forever and ever and ever.
If you don't fund, hire, and staff your support org with good people at multiple tiers, don't be surprised when you get crap. There just aren't enough good minds to service the incoming workload.
Oh, and for god's sake have a decent "to engineering" escalation path for the support folks.
Unfortunately, the support function is almost always a cost center and beholden to only, if anything, response time related SLAs. Instead of customer satisfaction, etc.
I was considering writing to Azure support about that...
My last billing issue with them took ~120 days to resolve, so I don't have much hope.
You are also right that it is across storage accounts, and access tiers: I've now looked at some of our other SAs, including ones in the Hot tier, and yep, they see the same bizarre uptick. Given that Hot tier is experiencing it, that rules out the possibility of "human error" w.r.t. to it not being a Cool-tier early deletion.
> Is this similar to what you're seeing ?
You've nailed it precisely; it sounds exactly like what we're seeing.
(What a small world HN is sometimes!)
¹the numbers on our end wiggle a bit, so I'm not sure about how precise I can estimate the uptick. But 70% is as close as any guess.
One of the most effective ways I've found to drive change in how product management and execs treat engineering was getting average salary numbers by type of position for my team out of HR, and attributing costs toward every major feature request + meetings in a weekly report.
My manager at the time, the CEO of a VC funded startup, did not see the point. We were after all already paying for the engineers, and they were always fully loaded. Until he'd seen a couple of reports, and saw what it told us about prioritisation.
Our project and product managers got a lot less meeting-happy after seeing one regular meeting they were very fond of cost an estimated 10k/month due to the number of people attending, and they got a lot more thoughtful about feature request and how to prioritize after seeing the real costs of supposedly "trivial" requests that had previously been slipped past normal processes.
Same applies on the cloud front, yet a lot of what has driven the adoption of cloud is that it often helps engineering escape cost analysis and efforts to optimize costs.
And so you get organisations which often have no idea which decisions contribute what to those costs.
I don't believe a baseline exists. The cloud is a new thing, and the fashion on how to use it is still changing constantly.
You also can not use this number as a proxy for any other thing. It's just what it is, you can't look for deeper connections.
Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. At one point we literally pulled all the billing data into a SQLite database. (And that was a great idea, as it made queries against it much faster.)
Further, most higher ups seem to have a sense of "it won't be invented here", and so a lot of stuff ends up in vendor land; need more people then to pull those separate bills into one spot, and start attributing costs against the revenue they might help generate.
I've also worked a fair number of places where people were cagey about telling eng how much they're spending where. If you want your IT dept. to cut costs, they have to know where the money is going.
And then there's the yak shaving of it all: we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Cool storage should be (price * amount stored) and the "how much are we storing" graph is not proportional to the cost graph. The only gotcha is an early delete penalty, and we are also pretty confident we're not hitting that. (We know what's being deleted when, and none of it should be subject to it.) But how does it take a over week to answer the question?