News for hackers. Stuff that matters.
Re:Introducing Gemma 3n by JohnKemeny on Thursday June 26, 2025 @06:36PM (#44392158)
Is that actually a useful benchmark, or is it just for the laughs? I've never really understood that.
Re:Introducing Gemma 3n by simonw on Thursday June 26, 2025 @07:30PM (#44392472)
It was supposed to be a joke. But weirdly it turns out there's a correlation between how good a model is and how good it as at my stupid joke benchmark.

I didn't realize quite how strong the correlation was until I put together this talk: https://simonwillison.net/2025/Jun/6/six-months-in-llms/

Re:Introducing Gemma 3n by moritzwarhier on Friday June 27, 2025 @03:58AM (#44394731)
Always loved this example, what do you think of ASCII art vs SVG?

Since it's not a formal encoding of geometric shapes, it's fundamentally different I guess, but it shares some challenges with the SVG tasks I guess? Correlating phrases/concepts with an encoded visual representation, but without using imagegen, that is.

Do you think that "image encoding" is less useful?

It's a thing I love to try with various models for fun, too.

Talking about illustration-like content, neither text-based ASCII art nor abusing it for rasterization.

The results have been interesting, too, but I guess it's less predictable than SVG.

Re:Introducing Gemma 3n by simonw on Friday June 27, 2025 @08:16AM (#44396194)
I've had disappointing results with ASCII art so far. Something I really like about SVG is that most models include comments, which give you an idea of what they were trying to do.
Re:Introducing Gemma 3n by moritzwarhier on Friday June 27, 2025 @01:17PM (#44398546)
Yes, the comments part makes sense, you also included it in the talk (I read the transcript but forgot to mention it in my comment, sorry :)

It makes sense, since it works adds associations between descriptions and individual shapes / paths etc., similar to other code.

Re:Introducing Gemma 3n by throw0363fc53 on Saturday June 28, 2025 @11:41AM (#44405499)
No way *wink*, our DevRel don't push for a good outcome on this one test case to get positive coverage on the top independent blog read by LLM people!

https://simonwillison.net/2025/May/20/google-io-pelican/

Re:Introducing Gemma 3n by OtherShrezzing on Thursday June 26, 2025 @06:54PM (#44392246)
For me, it shows if LLM are generalising from their training data. LLM understand all of the words in the prompt. they understand the spec for svg better than any human. They know what a bird is. They know what a bike is. They know how to draw (and given access to computer-use could probably ace this test). They can plan and execute on those plans.

Everything here should be trivial for LLM, but they’re quite poor at it because there’s almost no “how to draw complex shapes in svg” type content in their training set.

Re:Introducing Gemma 3n by mvdtnz on Thursday June 26, 2025 @07:10PM (#44392348)
It's a way for simonw to get hackernews upvotes and promote his blog. In that sense it's very useful.
Re:Introducing Gemma 3n by simonw on Thursday June 26, 2025 @07:31PM (#44392477)
Scored a whole two upvotes here, my scheme is clearly working great!
Re:Introducing Gemma 3n by DrammBA on Friday June 27, 2025 @12:28AM (#44393704)
Leave some upvotes for the rest of us
Re:Introducing Gemma 3n by Aeolun on Thursday June 26, 2025 @09:42PM (#44393098)
Kinda feel like the content is a much better reason to visit than the pelicans. Though I suppose the pelicans are part of the content.

I'm quite happy that there's someone with both the time to keep up with all the LLM/AI stuff, that is also good enough at writing amusing stuff that I want to keep reading it.

Re:Introducing Gemma 3n by mathgeek on Friday June 27, 2025 @04:42AM (#44395014)
> Kinda feel like the content is a much better reason to visit than the pelicans.

That's how the pelicans get ya.

Re:Introducing Gemma 3n by owebmaster on Thursday June 26, 2025 @08:47PM (#44392881)
I think in 5 years we might have some ultra-realistic pelicans and this benchmark will turn out quite interesting.
Re:Introducing Gemma 3n by lofaszvanitt on Friday June 27, 2025 @03:04AM (#44394377)
And then the author will try the "Pelican tries to swallow the capybara as-is". And it will fall apart again.
Re:Introducing Gemma 3n by moritzwarhier on Friday June 27, 2025 @03:59AM (#44394743)
It's this part where it gets interesting... how exactly it falls apart :D
Re:Introducing Gemma 3n by lofaszvanitt on Monday June 30, 2025 @02:26PM (#44426363)
like a quack
Re:Introducing Gemma 3n by jerpint on Friday June 27, 2025 @04:20AM (#44394867)
It’s been useful though given the authors popularity I suspect it’s only a matter of time new LLMs become “more aware” of it
Re:Introducing Gemma 3n by dominicrose on Friday June 27, 2025 @04:32AM (#44394952)
It's useful because it's SVG so it's different than other image generation methods.