mikeagb

I agree that stating outright that the answer is no is a bit too strong of a statement. The general consensus has definitely been that fine-tuning (especially instruction fine-tuning) is primarily to pick up style over facts, but that doesn't mean it's not doable. Continuous pre-training is used to instill new knowledge, and the line where it becomes "fine-tuning" rather from "continuous pre-training" is not obvious to me.

The question of how to fine-tune to teach LLMs facts/knowledge is definitely something we're interested in exploring more in future work. The common opinion seems to me at least to be that fine-tuning is more to teach the model how to use the knowledge it already has to complete a specific task rather than to instill new knowledge, and that RAG should be sued to provide more specific context. However, I personally believe there is potential in fine-tuning for "memorization" or learning, and am excited to see new developments in the field.