Re:Distilling Step-by-Step Outperforming Larger Language Mod...
by spuz on Thursday May 04, 2023 @05:49AM
(#35813120)
When you say prolog, are you referring to the programming language, or something else?
Re:Distilling Step-by-Step Outperforming Larger Language Mod...
by worldofideas123 on Thursday May 04, 2023 @06:17AM
(#35813309)
I was referring to the prolog programming language. What would happen if we replace the rationale with a concrete program in prolog: Example
tag = daughter(X), rationale = father(_,X),female(X); mother(_,X),female(X).
(2) Hypothesis: The success of distilling chain of thought learning is that it provides a pseudo large context window. The input pair (label,rationale) allows the student LLM mimic a large context window (the context window of a human agent that informs the system by providing the rationale data). So it reduces the embedding distance between words for which entailment is difficult to obtain with short context windows.
(3) The above hypothesis or explanation suggests to perform the training process in two phases: in the first phrase a smaller model is used as a teacher, this allow the learning model to connect d-distant words, in the second phrase the training process continue with new data using the rationale provided by a stronger larger LLM. In the first phase the learning system learn to infer d-distant word relations, in the second phase it is prepared to connect (2*d)-distant word because it have developed the necessary skills in the first phase.
This multi-phase approach could be tailored using three meta-parameters: the number of teachers, the relative power of each LLM teacher, and the percentage of data in the training set for each teacher.
(4) I would like that they would release for free the LLM that they have developed, those small but powerful LLM can provide individuals users to introduce themselves in the SOTA realm.
Edited: grammar and clarifying the ideas.