News for hackers. Stuff that matters.
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by pcwelder on Thursday January 02, 2025 @08:01AM (#42574044)
Was the comparison done with or without code context (as obtained using RAG or letting Sonnet ask for files)?
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by jeswin on Thursday January 02, 2025 @08:05AM (#42574068)
In the absense of other information, looks like a cherry-picked example to me.
Re:We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for ... by samatdav on Friday January 03, 2025 @12:05AM (#42582648)
We used a single file for the context. It is a cherry-picked example, you are right. I wanted to demonstrate a simple visual change that our model did correctly unlike Sonnet-3.5. Since we are just getting started, we don't have many features like making changes across multiple files in the code editor so it would be harder to demo. Our premise is that a smaller fine-tuned works better than a large, general-purpose SOTA model. We plan to share more metrics and data in the future.