News for hackers. Stuff that matters.
Re:Llama-3.3-70B-Instruct by Kiro on Friday December 06, 2024 @05:10PM (#42345084)
And how do you calculate the cost of that?
Re:Llama-3.3-70B-Instruct by nabakin on Friday December 06, 2024 @11:31PM (#42347282)
GPU cost ÷ GPU token throughput = cost per token, will get you close then you compare the cost per token between solutions.

To explain how to calculate it precisely, I would have to write a blog post. There are dozens of factors that go into it and they vary based on your use case like GPU type/setup, cloud provider, inferencing engine, context size, the minimum throughput and latency you'd be willing to have your users experience, LLM quantization, KV cache configuration, etc.

I'm sure there are cost analyses out there for Llama 3.1 70b you could find though.