Re:Adaptive LLM routing under budget constraints
by baq on Tuesday September 02, 2025 @02:12AM
(#45099654)
In which context? The serious engineering folks are still in exploration phase, costs is mostly not a concern as long as shipping velocity increases. Reselling repackaged tokens is a different beast, no experience here
Re:Adaptive LLM routing under budget constraints
by dahcryn on Tuesday September 02, 2025 @07:11AM
(#45101476)
LLM is far from the highest AI related cost, so we basically don't care about optimizing LLMs.
Obviously we don't use the super expensive ones like GPT4.5 or so. But we don't really bother with mini models, because GPT4.1 etc.. are cheap enough.
Stuff like speech to text etc.. are still way more expensive, and yes there we do focus on cost optimization. We have no large scale image generation use cases (yet)
So far, my experience has been that it's just too early for most people / applications to worry about cost - at most, I've seen AI to be accountable for 10% of cloud costs. But very curious if others have other experiences.