News for hackers. Stuff that matters.
Re:NanoGPT by CuriouslyC on Wednesday January 11, 2023 @09:07AM (#34338594)
This is hard a-priori, but fairly easy post-facto. Model distillation isn't a common practice yet, but it has already been demonstrated to be quite effective for specific use cases.
Re:NanoGPT by visarga on Wednesday January 11, 2023 @03:51PM (#34344554)
Distillation works but somehow we see very few papers doing it at this scale.