News for hackers. Stuff that matters.

A comment on ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations. Parent comment

Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu... by halflings on Wednesday January 08, 2020 @10:00PM (#21998159)
> Edit: I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model.

Huggingface did just that: https://medium.com/huggingface/distilbert-8cf3380435b5

[ Parent | on HN ]

There are no comments on this comment yet.