News for hackers. Stuff that matters.
Re:Gemma3 – The current strongest model that fits on a singl... by antirez on Wednesday March 12, 2025 @05:23AM (#43341252)
After reading the technical report do the effort of downloading the model and run it against a few prompts. In 5 minutes you understand how broken LLM benchmarking is.
Re:Gemma3 – The current strongest model that fits on a singl... by archerx on Wednesday March 12, 2025 @06:45AM (#43341782)
That's why I like giving it a real world test. For example take a podcast transcription and ask it to make show notes and summary. With a temperature of 0 different models will tackle the problem in different ways and you can infer if they really understood the transcript. Usually the transcripts that I give it come from about 1 hour of audio of two or more people talking.
Re:Gemma3 – The current strongest model that fits on a singl... by antirez on Wednesday March 12, 2025 @01:43PM (#43345754)
Good test. I'm slowly accumulating private tests that I use to rate LLMs, and this one was missing... Thanks.
Re:Gemma3 – The current strongest model that fits on a singl... by amelius on Wednesday March 12, 2025 @07:12AM (#43341993)
Aren't there any "blind" benchmarks?
Re:Gemma3 – The current strongest model that fits on a singl... by nathanasmith on Wednesday March 12, 2025 @09:39AM (#43343150)
Unfortunately that wouldn't help as much as you think since talented AI labs can just watch the public leaderboard and note what models move up and down to deduce and target whatever the hidden benchmark is testing.
Re:Gemma3 – The current strongest model that fits on a singl... by nickthegreek on Wednesday March 12, 2025 @11:46AM (#43344464)
OpenRouter Arena Ratings are probably the closet thing.
Re:Gemma3 – The current strongest model that fits on a singl... by toinewx on Wednesday March 12, 2025 @05:25AM (#43341258)
can you expand a bit?
Re:Gemma3 – The current strongest model that fits on a singl... by antirez on Wednesday March 12, 2025 @05:26AM (#43341264)
The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.
Re:Gemma3 – The current strongest model that fits on a singl... by anon373839 on Wednesday March 12, 2025 @06:53AM (#43341858)
I’d recommend trying it on Google AI Studio (aistudio.google.com). I am getting exceptional results on a handful of novel problems that require deep domain knowledge and structured reasoning. I’m not able to replicate this performance with Ollama, so I suspect something is a bit off.
Re:Gemma3 – The current strongest model that fits on a singl... by pzo on Wednesday March 12, 2025 @07:54AM (#43342256)
Maybe model is sensitive to quantization, by default ollama quantize it significantly.
Re:Gemma3 – The current strongest model that fits on a singl... by tarruda on Wednesday March 12, 2025 @10:32AM (#43343683)
I tried ollama fp16 and it had the same issues.
Re:Gemma3 – The current strongest model that fits on a singl... by tarruda on Wednesday March 12, 2025 @10:32AM (#43343674)
Same experience here: On AI Studio, this is easily one of the strongest models I have used, including when compared to proprietary LLMs.

But ollama and openwebui performance is very bad, even when running the FP16 version. I also tried to mirror some of AI studio settings (temp 1 and top p 0.95) but couldn't get it to produce anything useful.

I suspect there's some bug in the ollama releases (possibly wrong conversation delimiters?). If this is fixed, I will definitely start using Gemma 3 27b as my main model.

Re:Gemma3 – The current strongest model that fits on a singl... by anon373839 on Thursday March 13, 2025 @09:39AM (#43353239)
Update: Unsloth is recommending a temperature of 0.1, not 1.0, if using Ollama. I don’t know why Ollama would require a 10x lower value, but it definitely helped. I also read some speculation that there might be an issue with the tokenizer.
Re:Gemma3 – The current strongest model that fits on a singl... by alekandreev on Wednesday March 12, 2025 @06:35AM (#43341721)
Hey, Gemma engineer here. Can you please share reports on the type of prompts and the implementation you used?
Re:Gemma3 – The current strongest model that fits on a singl... by sgt101 on Wednesday March 12, 2025 @09:32AM (#43343071)
vibe testing, vibe model engineering...
Re:Gemma3 – The current strongest model that fits on a singl... by tarruda on Wednesday March 12, 2025 @10:34AM (#43343701)
Can you share the all the recommended settings to run this LLM? It is clear that the performance is very good when running on AI studio. If possible, I'd like to use the all the same settings (temp, top-k, top-p, etc) on Ollama. AI studio only shows Temperature, top-p and output length.
Re:Gemma3 – The current strongest model that fits on a singl... by antirez on Wednesday March 12, 2025 @01:45PM (#43345792)
Hello! I tried to show it Redis code yet not released (llama.cpp 4 bit quants and the official web interface) and V3 can reason about the design tradeoffs, but (very understandably) Gemma 3 can't. I also tried to make it write a simple tic tac toe Montecarlo program, and it didn't account for ties, while SOTA models consistently do.
Re:Gemma3 – The current strongest model that fits on a singl... by tarruda on Wednesday March 12, 2025 @05:39AM (#43341322)
In my experience, Gemma models were always bad at coding (but good at other tasks).
Re:Gemma3 – The current strongest model that fits on a singl... by kamranjon on Thursday March 13, 2025 @12:37AM (#43350443)
I really respect the work that you've done, but I am always very surprised when people just speak anecdotally as though it is truth with regards to AI models these days. It's as if everyone believes they are an expert now, but have nothing of substance to provide but their gut feelings.

It's as if people don't realize that these models are used for many different purposes, and subjectively one person could think one model is amazing and another person think it's awful. I just would hope that we could at least back up statements like "The model performs very poorly in practice" with actual data or at least some explanation of how it performed poorly.

Re:Gemma3 – The current strongest model that fits on a singl... by bearjaws on Wednesday March 12, 2025 @06:55AM (#43341869)
Prompt adherence is pretty bad from what I can tell.