Are OpenAI and Anthropic Losing Money on Inference?

Re:Are OpenAI and Anthropic Losing Money on Inference? by martinald on Thursday August 28, 2025 @06:32PM (#45057791)

Thanks for the correction (author here). I'll update the article - very fair point on compute on input tokens which I messed up. Tbh I'm pleased my napkin math was only 7x off the laws of physics :).

Even rerunning the math on my use cases with way higher input token cost doesn't change much though.

[ Parent | on HN ]

Re:Are OpenAI and Anthropic Losing Money on Inference? by chillee on Thursday August 28, 2025 @06:51PM (#45057928)

The 32 parallel sequences is also arbitrary and significantly changes your conclusions. For example, if they run with 256 parallel sequences then that would result in a 8x cheaper factor in your calculations for both prefill and decode.

The component about requiring long context lengths to be compute-bound for attention is also quite misleading.

[ Parent | on HN ]

Re:Are OpenAI and Anthropic Losing Money on Inference? by Barbing on Thursday August 28, 2025 @08:05PM (#45058427)

Anyone up to publishing their own guess range?

[ Parent | on HN ]