Coding with Llama 3.1, New DeepSeek Coder and Mistral Large

Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by Workaccount2 on Thursday July 25, 2024 @11:40AM (#41069944)

Is there a reason people seem to exclude google models? I see lots of talk about llama, mistral, deepseek, and then ofc gpt and claude, but not much about gemma and gemini despite them benchmarking well.

[ Parent | on HN ]

Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by anotherpaulg on Thursday July 25, 2024 @01:26PM (#41071152)

The main aider leaderboard includes gemini models. I benchmarked gemma-2-27b-it, but it did so badly that it wasn't worth adding to the leaderboard.

https://aider.chat/docs/leaderboards/

[ Parent | on HN ]

Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by Lash-L on Thursday July 25, 2024 @11:33PM (#41075566)

Interesting - why do you think that is?

can-ai-code seems to have Gemma 2-27b as pretty capable - beating out some of the other models that perform really well on aider: https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...

[ Parent | on HN ]

Re:Coding with Llama 3.1, New DeepSeek Coder and Mistral Large by 22c on Friday July 26, 2024 @12:20AM (#41075741)

Not an expert, but I believe it's related to this difference:

> LLMs which are good at editing code, not just good at writing code

So it may be that Gemma can write new code, but is not particularly good at editing or refactoring code.

[ Parent | on HN ]