News for hackers. Stuff that matters.

A comment on Cost Per 1M tokens Of Self Hosting Llama-3. Parent comment

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by veryrealsid on Friday June 14, 2024 @01:35PM (#40682872)
Interesting, I got very different results depending on how I ran the model, will definitely give this a try!

edit: Actually could you share how long it took to make a query? One of our issues is we need it to respond in a fast time frame

[ Parent | on HN ]
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by jezzarax on Friday June 14, 2024 @04:42PM (#40684905)
I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.
[ Parent | on HN ]