News for hackers. Stuff that matters.
Re:What We Know About LLMs (A Primer) by hallqv on Tuesday July 25, 2023 @03:14PM (#36867842)
RLHF is not part of LLaMa pretraining, or pretraning of any other models for that matter. RLHF comes after pretraining (which is just minimizing next token prediction loss). https://twitter.com/Jeande_d/status/1661833563069620247/phot...
Re:What We Know About LLMs (A Primer) by jsmith45 on Tuesday July 25, 2023 @04:53PM (#36869335)
Seems like a classic case of a term of art overlapping with normal English terminology.

Knowing that you will be doing further training on a provided model (even "just" extensive fine-tuning), one would want to distinguish the training done before you get your hands on it, from the training you do. An obvious word for that previous training is pre-training, which unfortunately conflicts with a term of art.

Re:What We Know About LLMs (A Primer) by jawerty on Tuesday July 25, 2023 @06:39PM (#36870686)
I see, that’s my misunderstanding I was grouping all training as pretraining
Re:What We Know About LLMs (A Primer) by wilhelm____ on Wednesday July 26, 2023 @08:39AM (#36876667)
pre-training is developing the language model's base understanding of conditional word probabilities.

SFT and RLHF is attempting to further guide the model in terms of steerability + alignment of output.

In fact, the InstructGPT authors were worried about losing the pre-trained model's underlying probability distribution, so they try a version where it penalizes the model deviating too significantly from the original distribution (using KL). I don't remember them seeing a significant difference in performance.