Fine tuning against in-house code seems like a small gain over a base model and search. It’s unlikely your code is unique and special and big enough that it’s hard to get results from a base model. You’ll be pinned to a certain version of a certain model, and you won’t be able to upgrade to future models nearly as quickly. Of course, you’re also fighting time again on each commit changing the code unless you continually fine tune it.
A RAG model might still struggle with a super vague question like “where does the foo cal bar with bax set” but it’s unlikely that this would work for fine tuning as well. This is where static code search by symbols really should be used.
Tbh the hardest part is the lifecycle - ie new data, updating, serving etc - that seems to be the biggest issue
A lot of financial, legal and health companies do fine-tuning! Reasoning finetuning via GRPO is also very powerful since you don't need any cot data in between! Just inputs and outputs!
Is it just a matter of assembling Q/A pairs like: “What’s class X?”, “class X { … }”
Do you really need to do this training on the base model instead, which means you have to fine tune chat on it afterward?
How does this work?
You will generally get better results when you fine-tune the base model on your data.
Since you still want to use it with the chat template in the end, you fine-tune the base model with the chat template with your specific data.
From there you'll have a lora that knows your data alright, but still doesn't really work for chatting.
You take that lora, merge it with the base model. Let's call this the stage model.
Then you use mergekit to merge the base model with both the stage model and the chat model. I used the TIES merge method in the past. Now you have your final model.
I use vLLM for inference, and needed access to multiple fine tunes on only a single set of hardware. So from that point I go and take the base model and my final model and extract a new lora. I also take the base model and chat model and extract another lora for that. Then I load up vLLM with the base model and as many of the fine tune loras I need + the chat lora.
The only time this hasn't worked is if the chat model adds a bunch of new tokens on top of the base model. If I remember right there was an issue with that
This has worked well for me in the past.
I've done it, 1/2 the team though it was great 20% of the time, 1/2 the team hated it from day 0. I used roughly 500K lines of code.
> How much effort is required to turn code into something one can use for fine-tuning?
Very little to moderate, less than 200 lines of python, QWEM FIM, HF, LLAMA.CPP, LLAMA.CPP code extension.
> RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach.
The only problem either way is keeping the information up to date, RAG just adds more cost to the inference process (which at my dev speed is pretty important).
> How much effort is required to turn code into something one can use for fine-tuning?
Fine tuning "fill in the middle" process is the process of taking a file, cutting out a some text in the middle and asking AI to guess what was there - there is a hugging face example that will have you doing it in an hour or less - your OPs team saying "No you cant litreally copy all code to a single folder" is probably the biggest hurdle (advise them you'll do it in CI and then they can stand up a FIM training endpoint that accepts a csv, pretty easy)
I know its coming but "mUlTi GpU PlZ" :pleading: <3
RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach.
How much effort is required to turn code into something one can use for fine-tuning?