News for hackers. Stuff that matters.

A comment on 70B Llama 2 at 35tokens/second on 4090. Parent comment

Re:70B Llama 2 at 35tokens/second on 4090 by BoorishBears on Wednesday September 13, 2023 @02:58PM (#37500297)
They made a technological advancement that allowed them to provide better output cheaper and faster and passed the savings onto their users...

Interesting to spin that into "not a good indicator".

[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by Palmik on Wednesday September 13, 2023 @03:21PM (#37500558)
Yes, and that's great, but what I meant is that it can definitely be much cheaper still, especially if you do not need a model that needs to be able to do everything, but is specialized on one or a few use cases.
[ Parent | on HN ]