Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4
by bfirsh on Tuesday September 12, 2023 @05:13PM
(#37488474)
Founder of Replicate here. It's early indeed.
OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B.
Our current models are far from efficient, but I'm surprised 7B turned out 100x more expensive. By our math, it's about ~10x more expensive right now for the base model. For fine-tuned models, we're roughly the same price. Rony – will message you on Discord to dig in more. :)
Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4
by yixu34 on Wednesday September 13, 2023 @02:05AM
(#37492776)
We're working on LLM Engine (https://llm-engine.scale.com) at Scale, which is our open source, self-hostable framework for open source LLM inference and fine-tuning. We have similar findings to Replicate: Llama 2 70B can be comparable to GPT 3.5 price, etc. Would be great to discuss this further!
Also, it's worth noting that Replicate started out with a focus on image generation, and their inference stack for LLMs is comically inefficient. A significant fraction of the 100x cost difference you mentioned can be made up by using an optimized inference server like vLLM. Replicate knows about this and is working hard on improving their stack, it's just really early for all of us. :)