This is how AWS made its fortune.
Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business.
And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
> And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.
They’re not saying they regret doing it, or that it was a mistake.
They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized.
Pretty standard business or life iteration.
It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations.
The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.
1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage.
2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money).
3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter.
4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions.
5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change.
6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates.
I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.
Happy to give you (or anyone here) some trial credits if interested! Email address in my profile.
What didn't happen is any analysis of cost/benefit up front. Many of the corporate decisions around AI have seemed characterized by companies blindly copying each other.
The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering.
Engineering payroll needed to be slashed and the C-Suite was happy to let their managers get some lashes in on the uppity workers.
The point of AI is:
1. It's bad engineering and everyone knows it.
2. Management is pushing it through anyway.
3. Deal with it, asshole.
The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control.
Edit: It is just as much about affirming the exalted status of management in comparison to labor. It's a bitter reinscription of the status hierarchy in tech; Management is good and labor is bad; Management is considerate and labor is ignorant; Management is strong and labor is weak; Management is superior and labor is inferior.
> The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control.
On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.