News for hackers. Stuff that matters.
Re:The last six months in LLMs, illustrated by pelicans on b... by simonw on Sunday June 08, 2025 @07:57AM (#44216412)
It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining.

I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models.

(Then I would also expand the judging panel to three vision LLMs from different model families which vote on each round... partly because it will be interesting to track cases where the judges disagree.)

I'm not sure if it's worth me doing that though since the whole "benchmark" is pretty silly. I'm on the fence.

Re:The last six months in LLMs, illustrated by pelicans on b... by ontouchstart on Sunday June 08, 2025 @08:37AM (#44216626)
Very nice talk, acceptable by general public and by AI agent as well.

Any concerns about open source “AI celebrity talks” like yours can be used in contexts that would allow LLM models to optimize their market share in ways that we can’t imagine yet?

Your talk might influence the funding of AI startups.

#butterflyEffect

Re:The last six months in LLMs, illustrated by pelicans on b... by threecheese on Sunday June 08, 2025 @09:25AM (#44216862)
I welcome a VC funded pelican … anything! Clippy 2.0 maybe?

Simon, hope you are comfortable in your new role of AI Celebrity.

Re:The last six months in LLMs, illustrated by pelicans on b... by demosthanos on Sunday June 08, 2025 @08:42AM (#44216643)
I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up.

Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...

Re:The last six months in LLMs, illustrated by pelicans on b... by throwaway31131 on Sunday June 08, 2025 @12:01PM (#44217764)
I’d say it doesn’t really matter. There is no universally good benchmark and really they should only be used to answer very specific questions which may or may not be relevant to you.

Also, as the old saying goes, the only thing worse than using benchmarks is not using benchmarks.

Re:The last six months in LLMs, illustrated by pelicans on b... by diggan on Sunday June 08, 2025 @09:39AM (#44216939)
Yeah, this is the problem with benchmarks where the questions/problems are public. They're valuable for some months, until it bleeds into the training set. I'm certain a lot of the "improvements" we're seeing are just benchmarks leaking into the training set.
Re:The last six months in LLMs, illustrated by pelicans on b... by travisgriggs on Sunday June 08, 2025 @10:52AM (#44217361)
That’s ok, once bicycle “riding” pelicans become normative, we can ask it for images of pelicans humping bicycles.

The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible. A plausibility machine (LLM) will struggle with the implausible, until it can abstract well.

Re:The last six months in LLMs, illustrated by pelicans on b... by diggan on Sunday June 08, 2025 @11:54AM (#44217725)
> The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible

Until there is enough unique/new subject-verb-objects examples/benchmarks so the trained model actually generalized it just like you did. (Public) Benchmarks needs to constantly evolve, otherwise they stop being useful.

Re:The last six months in LLMs, illustrated by pelicans on b... by demosthanos on Sunday June 08, 2025 @12:57PM (#44218087)
To be fair, once it does generalize the pattern then the benchmark is actually measuring something useful for deciding if the model will be able to product a subject-verb-object SVG.
Re:The last six months in LLMs, illustrated by pelicans on b... by zahlman on Sunday June 08, 2025 @01:10PM (#44218176)
I can't fathom this working, simply because building a model that relates the word "ride" to "hump" seems like something that would be orders of magnitude easier for an LLM than visualizing the result of SVG rendering.
Re:The last six months in LLMs, illustrated by pelicans on b... by 6LLvveMx2koXfwn on Sunday June 08, 2025 @10:02AM (#44217070)
I would definitely say he had no intention of doing that and was doubling down on the original joke.
Re:The last six months in LLMs, illustrated by pelicans on b... by colecut on Sunday June 08, 2025 @10:27AM (#44217214)
The road to hell is paved with the best intentions

clarification: I enjoyed the pelican on a bike and don't think it's that bad =p

Re:The last six months in LLMs, illustrated by pelicans on b... by telotortium on Sunday June 08, 2025 @07:31PM (#44220149)
Yeah, Simon needs to release a new benchmark under a pen name, like Stephen King did with Richard Bachman.
Re:The last six months in LLMs, illustrated by pelicans on b... by Breza on Friday June 13, 2025 @11:09AM (#44269240)
Re:The last six months in LLMs, illustrated by pelicans on b... by fzzzy on Sunday June 08, 2025 @08:46AM (#44216659)
Even if it is a joke, having a consistent methodology is useful. I did it for about a year with my own private benchmark of reasoning type questions that I always applied to each new open model that came out. Run it once and you get a random sample of performance. Got unlucky, or got lucky? So what. That's the experimental protocol. Running things a bunch of times and cherry picking the best ones adds human bias, and complicates the steps.
Re:The last six months in LLMs, illustrated by pelicans on b... by simonw on Sunday June 08, 2025 @08:48AM (#44216671)
It wasn't until I put these slides together that I realized quite how well my joke benchmark correlates with actual model performance - the "better" models genuinely do appear to draw better pelicans and I don't really understand why!
Re:The last six months in LLMs, illustrated by pelicans on b... by johnrob on Sunday June 08, 2025 @11:41AM (#44217634)
Well, the most likely single random sample would be a “representative” one :)
Re:The last six months in LLMs, illustrated by pelicans on b... by pama on Sunday June 08, 2025 @09:01AM (#44216743)
How did the pelicans of point releases of V3 and of R1 (R1-0528) do compared to the original versions of the models?
Re:The last six months in LLMs, illustrated by pelicans on b... by tuananh on Sunday June 08, 2025 @09:48AM (#44216984)
until they start targeting this benchmark
Re:The last six months in LLMs, illustrated by pelicans on b... by simonw on Sunday June 08, 2025 @10:17AM (#44217175)
Right, that was the closing joke for the talk.
Re:The last six months in LLMs, illustrated by pelicans on b... by jonstewart on Sunday June 08, 2025 @02:13PM (#44218596)
It is funny to think that a hundred years in the future there may be some vestigial area of the models’ networks that’s still tuned to drawing pelicans on bicycles.
Re:The last six months in LLMs, illustrated by pelicans on b... by MichaelZuo on Sunday June 08, 2025 @09:26AM (#44216867)
I imagine the straightforward reason is that the “better” models are in fact significantly smarter in some tangible way, somehow.
Re:The last six months in LLMs, illustrated by pelicans on b... by more-nitor on Sunday June 08, 2025 @09:45AM (#44216968)
I just don't get the fuss from the pro-LLM people who don't want anyone to shame their LLMs...

people expect LLMs to say "correct" stuff on the first attempt, not 10000 attempts.

Yet, these people are perfectly OK with cherry-picked success stories on youtube + advertisements, while being extremely vehement about this simple experiment...

...well maybe these people rode the LLM hype-train too early, and are desperate to defend LLMs lest their investment go poof?

obligatory hype-graph classic: https://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Ga...

Re:The last six months in LLMs, illustrated by pelicans on b... by og_kalu on Sunday June 08, 2025 @03:02PM (#44218895)
Re:The last six months in LLMs, illustrated by pelicans on b... by Breza on Friday June 13, 2025 @11:11AM (#44269263)
Another advantage is you can easily include deprecated models in your comparisons. I maintain our internal LLM rankings at work. Since the prompts have remained the same, I can do things like compare the latest Gemini Pro to the original Bard.
Re:The last six months in LLMs, illustrated by pelicans on b... by dilap on Sunday June 08, 2025 @11:13AM (#44217487)
Joke or not, it still correlates much better with my own subjective experiences of the models than LM Arena!
Re:The last six months in LLMs, illustrated by pelicans on b... by Breza on Friday June 13, 2025 @11:04AM (#44269196)
I'd be really interested in evaluating the evaluations of different models. At work, I maintain our internal LLM benchmarks for content generation. We've always used human raters from MTurk, and the Elo rankings generally match what you'd expect. I'm looking at our options for having LLMs do the evaluating.

In your case, it would be neat to have a bunch of different models (and maybe MTurk) pick the winners of each head-to-head matchup and then compare how stable the Elo scores are between evaluators.