I agree that stating outright that the answer is no is a bit too strong of a statement. The general consensus has definitely been that fine-tuning (especially instruction fine-tuning) is primarily to pick up style over facts, but that doesn't mean it's not doable. Continuous pre-training is used to instill new knowledge, and the line where it becomes "fine-tuning" rather from "continuous pre-training" is not obvious to me.

This line between fine tuning and continuous pre-training is what I’m interested in. What is the investment difference between fine tuning, contoured-training and training from scratch? Do you (or anyone else) have any good sources or know of good examples where continuous pre-training is being done?

> There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition

The answer for someone asking that question is a strict no. Many people asking this stuff only have access to SFT, so it's a super no for them.

Honestly I don't get this weird obsession right now with LLMs and throwing random roadblocks in any sort of common knowledge of the subject. If someone in CS 101 asked if they could write a game engine in CSS you wouldn't get people lining up to tell them the answer isn't "No." despite it technically being possible (https://github.com/brookjordan/css-game-engine) because we understand that sometimes to enable understanding of a subject you need to setup some solid ground for new entrants to stand on.

Fine-tuning is not for knowledge. If you get comfortable enough to start experimenting with that application, you'll understand that there's some nuance to that statement either way and get to research/tinker/push boundaries armed with enough knowledge to not accept the simple no.

Funny, as a corollary to how impractical a game engine would be in CSS: my first pass over the text had me reading CSS as something like C Sharp Sharp. A non-existent language that, according to my brain, still seems like a more likely language to build the game engine in

Or, in other words, you were hallucinating, in the LLM sense.

What a fool believes, he sees.

https://en.wikipedia.org/wiki/Predictive_coding