Character.ai shows how to reduce LLM costs by 13 times

Posted by Oksitaine on Thursday July 04, 2024 @02:14PM
from the blog.character.ai dept.

https://blog.character.ai/optimizing-ai-inference-at-character-ai/

( 4 comments | 3 points | on HN )

Re:Character.ai shows how to reduce LLM costs by 13 times by gnabgib on Thursday July 04, 2024 @02:18PM (#40876941)

Article title: Optimizing AI Inference

Discussion (63 points, 14 days ago) https://news.ycombinator.com/item?id=40739225

[ Parent | on HN ]

Re:Character.ai shows how to reduce LLM costs by 13 times by uberman on Thursday July 04, 2024 @02:32PM (#40877035)

If I reduce the cost of something by 1x it becomes free...

[ Parent | on HN ]

Re:Character.ai shows how to reduce LLM costs by 13 times by Oksitaine on Thursday July 04, 2024 @02:41PM (#40877089)

Yes it's true, i mean he LLM costs for Character.ai are 13 times cheaper (thanks to their optimization with KV cache) than if they had simply used the API of an LLM service like Anthropic or ChatGPT.

[ Parent | on HN ]

Re:Character.ai shows how to reduce LLM costs by 13 times by uberman on Thursday July 04, 2024 @03:46PM (#40877531)

I think it is a great optimization. My point is that they can reduce the cost by 92% which is less than 1x (a fantastic optimization) but if they reduce the cost by 1x (100%) it would mean that the service would be free. Reducing the cost by 13x means that for every dollar I spend they would give me 13 dollars. This is a not uncommon error and i would not mention it, but it is the direct claim in the heading.

[ Parent | on HN ]