Ask HN: What's your biggest LLM cost multiplier?

Re:Ask HN: What's your biggest LLM cost multiplier? by zhug3 on Saturday January 31, 2026 @01:38PM (#46839342)

In my experience the biggest multiplier isn't any single variable it's the interaction between them. Fanout × retries × context growth compounds in ways that linear cost models completely miss.

The fix that worked for us: treat budget as a hard constraint, not a target. When you're approaching limit, degrade gracefully (shorter context, fewer tool calls, fallback to smaller model) rather than letting costs explode and cleaning up later.

Also worth tracking: the 90th percentile request often costs 10x the median. A handful of pathological queries can dominate your bill. Capping max tokens per request is crude but effective.

[ Parent | on HN ]

Re:Ask HN: What's your biggest LLM cost multiplier? by teilom on Saturday January 31, 2026 @04:34PM (#46841088)

+1 on interaction terms + tails : fanout × retries × context growth is where linear token math dies.

One thing we do in enzu is make “budget as constraint” executable: we clamp `max_output_tokens` from the budget before the call, and in multi-step/RLM runs we adapt output caps downward as the budget depletes (so it naturally gets shorter/cheaper instead of spiraling). When token counting is unavailable we explicitly enter a “budget degraded” mode rather than pretending estimates are exact.

Also agree p90/p95 cost/run matters more than averages; max-output caps are crude but effective.

Docs: https://github.com/teilomillet/enzu/blob/main/docs/PROD_MULT... and https://github.com/teilomillet/enzu/blob/main/docs/BUDGET_CO...

[ Parent | on HN ]