News for hackers. Stuff that matters.

A comment on Coding with Llama 3.1, New DeepSeek Coder and Mistral Large. Parent comment

Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by anotherpaulg on Thursday July 25, 2024 @01:26PM (#41071152)
The main aider leaderboard includes gemini models. I benchmarked gemma-2-27b-it, but it did so badly that it wasn't worth adding to the leaderboard.

https://aider.chat/docs/leaderboards/

[ Parent | on HN ]
Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by Lash-L on Thursday July 25, 2024 @11:33PM (#41075566)
Interesting - why do you think that is?

can-ai-code seems to have Gemma 2-27b as pretty capable - beating out some of the other models that perform really well on aider: https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...

[ Parent | on HN ]
Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by 22c on Friday July 26, 2024 @12:20AM (#41075741)
Not an expert, but I believe it's related to this difference:

> LLMs which are good at editing code, not just good at writing code

So it may be that Gemma can write new code, but is not particularly good at editing or refactoring code.

[ Parent | on HN ]