News for hackers. Stuff that matters.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by jezzarax on Friday June 14, 2024 @04:42PM (#40684905)
I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.

There are no comments on this comment yet.