Re:Tiny Language Models Come of Age
by gwern on Sunday October 08, 2023 @06:32PM
(#37815019)
> Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance.
I would, but they don't say what their dataset is that I can find anywhere, and the only thing they say about their instruction-tuned is that it's trained on 'publicly available' datasets. You know, the ones where a lot of them turn out under the hood to be drawing from the OA API or other pretrained models in some way or another...
> Especially not with pre-filtered/classified pre-training data.
Indeed not! But what exactly is prefiltering or classifying all that data...?
I would, but they don't say what their dataset is that I can find anywhere, and the only thing they say about their instruction-tuned is that it's trained on 'publicly available' datasets. You know, the ones where a lot of them turn out under the hood to be drawing from the OA API or other pretrained models in some way or another...
> Especially not with pre-filtered/classified pre-training data.
Indeed not! But what exactly is prefiltering or classifying all that data...?