Re:Learnings from fine-tuning LLM on my Telegram messages
by ojosilva on Tuesday November 28, 2023 @05:19AM
(#38444294)
It caught mine too. I'm weighting several alternatives to "fine-tuning model fine-tuning", meaning the back-and-forth, trial-and-error previous to massively running the full training set.
My goal is to fine-tune a model on our codebase. I find RAG to be too orthopedic, I'd really would like to train the model on what is each part of the code and how we do things and see how it responds with a more complete perspective that goes beyond context.
The options I've considered for pre-fine-tuning:
- using a service like vast.ai, runpod, gradient or similar
- use Google Collab
- getting a more powerful MacBook, M3max with plenty of RAM
My goal is to fine-tune a model on our codebase. I find RAG to be too orthopedic, I'd really would like to train the model on what is each part of the code and how we do things and see how it responds with a more complete perspective that goes beyond context.
The options I've considered for pre-fine-tuning:
- using a service like vast.ai, runpod, gradient or similar
- use Google Collab
- getting a more powerful MacBook, M3max with plenty of RAM