News for hackers. Stuff that matters.
Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4 by kcorbitt on Tuesday September 12, 2023 @02:39PM (#37485645)
Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one exception being for organizations where data security is non-negotiable and they really need to host on-prem. The calculus changes drastically though when you bring fine-tuning in, which lets a much smaller model outperform a larger one on many classes of task.

Also, it's worth noting that Replicate started out with a focus on image generation, and their inference stack for LLMs is comically inefficient. A significant fraction of the 100x cost difference you mentioned can be made up by using an optimized inference server like vLLM. Replicate knows about this and is working hard on improving their stack, it's just really early for all of us. :)

Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4 by bfirsh on Tuesday September 12, 2023 @05:13PM (#37488474)
Founder of Replicate here. It's early indeed.

OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B.

Our current models are far from efficient, but I'm surprised 7B turned out 100x more expensive. By our math, it's about ~10x more expensive right now for the base model. For fine-tuned models, we're roughly the same price. Rony – will message you on Discord to dig in more. :)

Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4 by yixu34 on Wednesday September 13, 2023 @02:05AM (#37492776)
We're working on LLM Engine (https://llm-engine.scale.com) at Scale, which is our open source, self-hostable framework for open source LLM inference and fine-tuning. We have similar findings to Replicate: Llama 2 70B can be comparable to GPT 3.5 price, etc. Would be great to discuss this further!
Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4 by Dowwie on Wednesday September 13, 2023 @01:14PM (#37499077)
How heavy of a lift is it to optimize inference?