News for hackers. Stuff that matters.

A comment on Refact LLM: New 1.6B code model reaches 32% HumanEval and is SOTA for the size. Parent comment

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by riku_iki on Monday September 04, 2023 @01:20PM (#37382612)
> almost reaches the same HumanEval

how can you tell that HumanEval is not leaked to your training data in some form?

[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by mityamitya on Monday September 04, 2023 @01:36PM (#37382841)
Hi! We ran LSH filtering over datasets to remove all code that can be similar to HumanEval samples.
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by riku_iki on Monday September 04, 2023 @02:10PM (#37383226)
so, we have to trust your procedure..
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by JegernOUTT on Tuesday September 05, 2023 @07:29AM (#37390314)
It can be checked if the model predicts canonical solutions from humaneval. I understand it is not ideal, but at least you can check it yourself

There are a bunch of other benchmarks too, check out the page https://huggingface.co/smallcloudai/Refact-1_6B-fim

Also, feel free to run any new benchmarks

[ Parent | on HN ]