Re:Mistral Fine-Tune
by newswasboring on Saturday May 25, 2024 @07:36AM
(#40474271)
What is your strategy for this? Do you finetune when a new flagship model is made available? You said local first, so I'm guessing you might have finetuned llama. But there are llama fine-tunes available which have better performance than the base model. How do you choose?
Re:Mistral Fine-Tune
by probably_wrong on Saturday May 25, 2024 @09:08AM
(#40474822)
Our strategy is to take a well-known, battle-tested model as a base, train it, and then hopefully one day release the fine-tuned model on HuggingFace.
Other than that, fine-tunes don't really matter for us because not many people are rushing to beat the top models on (say) Georgian POS tagging or Urdu sentiment analysis.
As long as the model can turn language into a reasonable vector, we're happy with it.
I have non-English human data annotated in a format that was designed for a very specific health-related study. LLMs haver never seen these annotations, non-English LLMs are not a top priority for companies, and we can only use offline-first ones for data privacy reasons.
In this scenario fine-tuning a general purpose LM works wonders.