News for hackers. Stuff that matters.

A comment on Refact LLM: New 1.6B code model reaches 32% HumanEval and is SOTA for the size.

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by umutisik on Monday September 04, 2023 @01:16PM (#37382559)
The title is misleading This model is not "SOTA for the size", there are smaller models that do 10-18% better in absolute score. The text says it's SOTA "among similar models" where they probably compare with other models with permissive licensing.
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by OlegKlimov1337 on Monday September 04, 2023 @01:35PM (#37382828)
AFAIK There is only one model that do better, it’s phi-1 and it’s python only, and it does not support fill-in-the-middle so you can't really use it.
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by umutisik on Monday September 04, 2023 @03:32PM (#37384055)
Phi-1-small also scores higher with 350M parameters. It helps to be specific about what the comparison is against when claiming SOTA.
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by mrob on Monday September 04, 2023 @01:29PM (#37382734)
"Permissive" usually refers to Free Software or Open Source licenses without copyleft requirements. OpenRAIL is a proprietary license because it imposes usage restrictions, contrary to both the Free Software and Open Source definitions.
[ Parent | on HN ]