Re:Cost Per 1M tokens Of Self Hosting Llama-3
by elorant on Friday June 14, 2024 @04:29PM
(#40684788)
Is this at 4-bit quantization? And how many tokens per second is the output?
[ Parent
| on HN ]
Re:Cost Per 1M tokens Of Self Hosting Llama-3
by hehdhdjehehegwv on Saturday June 15, 2024 @01:07AM
(#40687638)
I’m doing non-interactive tasks, but in terms of the A6000 running llama3 70b in chat mode it’s as usable as any of the commercial offerings in terms of speed. I read quickly and it’s faster than I read.