Re:GPT-3.5 Turbo fine-tuning and API updates
by armini on Tuesday August 22, 2023 @09:35PM
(#37230835)
How are you currently hosting your LLaMA 2? Any tips, tricks or advice?
[ Parent
| on HN ]
Re:GPT-3.5 Turbo fine-tuning and API updates
by syntaxing on Tuesday August 22, 2023 @10:35PM
(#37231249)
It depends on your needs. For instance, do you want to host an API or do you want to have a front end like chatGPT? Chances are, text-generation-webui [1] should get you pretty close to hosting it yourself. You simply clone the repo, download the model from huggingface using the included helper (download-model.py) and fire up the server with server.py. You can connect to it by SSH port tunneling on port 7860 (there's other way like Ngrok but SSH tunneling is the easiest and secure).
As for hosting, I found that runpod [2] has been the cheapest (not affiliated, just a user). All the other services tend to add up more than them when you include bandwidth and storage. There's some tutorials online [3] but a lot of them use the quantized version. You should be able to fit the original 70B with "load_in_8bit" on one A100 80GB.
[1] https://github.com/oobabooga/text-generation-webui [2] https://www.runpod.io/ [3] https://gpus.llm-utils.org/running-llama-2-on-runpod-with-oo...
Re:GPT-3.5 Turbo fine-tuning and API updates
by robertnishihara on Wednesday August 23, 2023 @12:46AM
(#37231980)
If you want to query the Llama-2 models, you can use Anyscale Endpoints [1]. Note: I work on this :)
Llama-2-70B is $1 / million tokens, which is the most cost-efficient on the market that I'm aware of.
Re:GPT-3.5 Turbo fine-tuning and API updates
by zo1 on Wednesday August 23, 2023 @01:41AM
(#37232249)
Can we supply our own fine-tuned models?
Re:GPT-3.5 Turbo fine-tuning and API updates
by andrewmunn on Wednesday August 23, 2023 @01:03AM
(#37232069)
How do you keep the cost down?