Re:GPT-3.5 Turbo fine-tuning and API updates
by robertnishihara on Wednesday August 23, 2023 @12:46AM
(#37231980)
If you want to query the Llama-2 models, you can use Anyscale Endpoints [1]. Note: I work on this :)
Llama-2-70B is $1 / million tokens, which is the most cost-efficient on the market that I'm aware of.
Re:GPT-3.5 Turbo fine-tuning and API updates
by zo1 on Wednesday August 23, 2023 @01:41AM
(#37232249)
Can we supply our own fine-tuned models?
Re:GPT-3.5 Turbo fine-tuning and API updates
by andrewmunn on Wednesday August 23, 2023 @01:03AM
(#37232069)
How do you keep the cost down?
As for hosting, I found that runpod [2] has been the cheapest (not affiliated, just a user). All the other services tend to add up more than them when you include bandwidth and storage. There's some tutorials online [3] but a lot of them use the quantized version. You should be able to fit the original 70B with "load_in_8bit" on one A100 80GB.
[1] https://github.com/oobabooga/text-generation-webui [2] https://www.runpod.io/ [3] https://gpus.llm-utils.org/running-llama-2-on-runpod-with-oo...