News for hackers. Stuff that matters.

A comment on We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation.

Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by prmoustache on Thursday January 02, 2025 @10:37AM (#42575334)
How do you measure code generation accuracy? Are there some base tests and if so how can I ensure the models aren't tuned for those tests only the same way vw cheated the emissions tests on their diesels?
[ Parent | on HN ]
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by samatdav on Thursday January 02, 2025 @11:50PM (#42582560)
We run a set of change requests on the discourse repo. Good point, we plan to publish more detailed testing benchmarks and metrics on the website.
[ Parent | on HN ]