Re:70B Llama 2 at 35tokens/second on 4090
by Tenoke on Wednesday September 13, 2023 @05:20AM
(#37494272)
You can also fine-tune gpt-3.5 though.
Re:70B Llama 2 at 35tokens/second on 4090
by Palmik on Wednesday September 13, 2023 @05:26AM
(#37494335)
Fine tuned gpt-3.5 is ~8x the price compared to regular gpt-3.5. It might be a good way to collect better training data though. :)
Re:70B Llama 2 at 35tokens/second on 4090
by BoorishBears on Wednesday September 13, 2023 @06:40AM
(#37494925)
That's absolute dirt cheap. It's like 1/10th the price fine-tuned GPT-3 was launched at, and funnily enough cheaper than many of the original embeddings endpoints.
Re:70B Llama 2 at 35tokens/second on 4090
by Palmik on Wednesday September 13, 2023 @07:25AM
(#37495267)
It's great that the costs are coming down, but it's still not economical for many activities.
The fact that some things had extreme margins before, and now they have less extreme margins, isn't a really good indicator.
Re:70B Llama 2 at 35tokens/second on 4090
by BoorishBears on Wednesday September 13, 2023 @02:58PM
(#37500297)
They made a technological advancement that allowed them to provide better output cheaper and faster and passed the savings onto their users...
Interesting to spin that into "not a good indicator".
Re:70B Llama 2 at 35tokens/second on 4090
by Palmik on Wednesday September 13, 2023 @03:21PM
(#37500558)
Yes, and that's great, but what I meant is that it can definitely be much cheaper still, especially if you do not need a model that needs to be able to do everything, but is specialized on one or a few use cases.
Things that require consistency: e.g. you want the chat / output to have certain "personality", consistent level of conciseness or formatting.
Things where examples are hard to fit into prompt: e.g. summarization, or other longer form tasks.
High volume, simpler tasks: Various data extraction tasks.
Two of my side projects (links in bio) use AI for summarization, and indeed consistency is a big issue there.