Cost Per 1M tokens Of Self Hosting Llama-3

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by jezzarax on Friday June 14, 2024 @12:50PM (#40682417)

llama.cpp + llama-3-8b in Q8 run great on a single T4 machine. Cannot remember the TPS I got there, but it was much above 6 mentioned in the article.

[ Parent | on HN ]

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by veryrealsid on Friday June 14, 2024 @01:35PM (#40682872)

Interesting, I got very different results depending on how I ran the model, will definitely give this a try!

edit: Actually could you share how long it took to make a query? One of our issues is we need it to respond in a fast time frame

[ Parent | on HN ]

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by jezzarax on Friday June 14, 2024 @04:42PM (#40684905)

I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.

[ Parent | on HN ]