Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu...
by samcodes on Wednesday January 08, 2020 @12:53PM
(#21993885)
> I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model.
IME this is the way to go; take an ensemble of big, accurate models, then distill them down to the smallest model you can get away with. There are really good tricks in this paper
IME this is the way to go; take an ensemble of big, accurate models, then distill them down to the smallest model you can get away with. There are really good tricks in this paper
https://www.aclweb.org/anthology/D19-5632/