News for hackers. Stuff that matters.
Re:Phi-3 Technical Report by ankit219 on Tuesday April 23, 2024 @05:57AM (#40130216)
Not trying to disparage them, but their models always give a feeling that it is overfitted on benchmarks hence they perform so well. On everyday tasks, it's much worse - chat or simple completion tasks.

Distilling can work and there are papers which suggest it does, but we still do not have a reliable mechanism which can distill knowledge from larger teacher models to smaller student models.

Re:Phi-3 Technical Report by moffkalast on Tuesday April 23, 2024 @11:14AM (#40132878)
This was the case for Phi-2, it was notoriously rubbish in practical use.