You probably don’t need to fine-tune LLMs

Re:You probably don’t need to fine-tune LLMs by rafaelero on Friday August 18, 2023 @10:25AM (#37176067)

I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need billions of tokens to make a LLM reason based on your own dataset. And even if they did acquire these reasoning skills, they probably wouldn't beat GPT-4. And we are not even getting into the costs of self-hosting these LLM's. So why bother? Just use an API from these companies with powerful models and tweak it to deal with your own need.

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by CharlesW on Friday August 18, 2023 @10:55AM (#37176538)

Did you try a Retrieval Augmented Generation (RAG) approach? If so, what made you give it up in favor of fine-tuning?

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by rafaelero on Friday August 18, 2023 @11:26AM (#37176915)

This is the approach I am using right now. My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. There is also the problem with context window limit; sometimes I have to truncate the relevant documents, limiting the capacity of the model to offer a good answer.

But I am very impressed with GPT-4's abilities in finding the correct answer just by reading those documents, so I think the only problem still enduring is the context window size.

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by CharlesW on Friday August 18, 2023 @11:48AM (#37177320)

> My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so.

That was my impression as well, which is why your comment was so interesting to me. Have you found tools/projects for the DAG approach that you'd recommend?

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by ZunarJ5 on Friday August 18, 2023 @11:00AM (#37176620)

https://github.com/khoj-ai/khoj

This is the easiest I found, on here too.

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by nico on Friday August 18, 2023 @12:20PM (#37177891)

The readme says it’s offline and doesn’t mention any LLMs

Does it not use any LLMs, or does it have its own, or how how does it work?

Thank you

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by ZunarJ5 on Friday August 18, 2023 @12:30PM (#37178074)

You can set it up so that it downloads Llama for offline use in the UI. You do not have to do it manually.

[ Parent | on HN ]

Re:You probably don’t need to fine-tune LLMs by clharman on Friday August 18, 2023 @08:28PM (#37184061)

@rafaelero I'm working on a blogpost (https://colinharman.substack.com/) to demonstrate this fact since I get a lot of tiresome questions like "why don't you just train instead of retrieving"

Do you have any scripts you could share for the training/eval process? Would love to credit you in the post

[ Parent | on HN ]