It's referred to as distillation in the literature in separate sub-fields (e.g. LRspeech, a paper about low-resource speech-to-text & text-to-speech from Microsoft Research, 2020, uses datasets generated by a large model to train a small one and refers to this as knowledge distillation: https://arxiv.org/abs/2008.03687 )

Matching logits is merely a special case of knowledge distillation. See section 2.1 https://arxiv.org/abs/1503.02531

Literally retraining on inputs and outputs without doing anything clever, does not to me seem to deserve quite such a fancy name. How do we end up with a smaller model if the dataset formats are exactly similar, except in some other special case, like using much more generated data to train the smaller model? Of course there are other deservedly clever things you could do besides matching logits, that would enable a smaller student model.

Here, the Stanford authors are not doing anything clever to enable a smaller model. They're just yoinking the fine-tuning onto what happens to be a smaller model. The destination model being smaller is not the point. The cheap yoinking of just the instruction tuning is the point. They used a small model as the destination because that was cheaper.

Sure, but the value of logit matching is that it allows the smaller model to access richer representations that it has sufficient capacity to express but insufficient capacity to learn. In this case, we're not trying to distill the full capabilities of the large model. We just want to mimic instruction following, and we don't have any reason to believe the smaller model has insufficient capacity. So doesn't the lesson of knowledge distillation tell us that this should work just by matching low temperature logits (in other words, matching argmax only)? I don't see how this is fundamentally different from the standard knowledge distillation setting - it's just an easier instance of it

1) It is a general version of knowledge distillation. For example, this paper from 2016 describes the same technique: Sequence-Level Knowledge Distillation [0]

> This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network

2) Fine-tuning is a step in the training process. Language models are first pre-trained, then fine-tuned. This is a pedantic quibble.

3) It is unsurprising that you don't understand ad hominem. Giving background information and pointing out the style of writing is relevant to arguments made.

[0] https://arxiv.org/abs/1606.07947

It's arguable that saying "EY has a very shallow understanding of ML" is even lower than ad hominem (which is DH1) on the pg scale [4], since pg specifically gives "The author is a self-important dilettante." as an example of DH0.

[4] http://www.paulgraham.com/disagree.html

>> the mask of the shoggoth

I threw a null cultural reference exception on that one. Can you please clarify?

Edit: I know the "Masks of Nyarlathotep" and what a shoggoth is; I just don't know what is the "mask of the shoggoth".

Edit 2: Really, the one thing I wanted to be able to do with a language generator has always been to get it to GM a CoC campaign for me. It's still the case that there's no chance of that. Just as an aside.