News for hackers. Stuff that matters.
Re:The last six months in LLMs, illustrated by pelicans on b... by simonw on Sunday June 08, 2025 @08:48AM (#44216671)
It wasn't until I put these slides together that I realized quite how well my joke benchmark correlates with actual model performance - the "better" models genuinely do appear to draw better pelicans and I don't really understand why!
Re:The last six months in LLMs, illustrated by pelicans on b... by johnrob on Sunday June 08, 2025 @11:41AM (#44217634)
Well, the most likely single random sample would be a “representative” one :)
Re:The last six months in LLMs, illustrated by pelicans on b... by pama on Sunday June 08, 2025 @09:01AM (#44216743)
How did the pelicans of point releases of V3 and of R1 (R1-0528) do compared to the original versions of the models?
Re:The last six months in LLMs, illustrated by pelicans on b... by tuananh on Sunday June 08, 2025 @09:48AM (#44216984)
until they start targeting this benchmark
Re:The last six months in LLMs, illustrated by pelicans on b... by simonw on Sunday June 08, 2025 @10:17AM (#44217175)
Right, that was the closing joke for the talk.
Re:The last six months in LLMs, illustrated by pelicans on b... by jonstewart on Sunday June 08, 2025 @02:13PM (#44218596)
It is funny to think that a hundred years in the future there may be some vestigial area of the models’ networks that’s still tuned to drawing pelicans on bicycles.
Re:The last six months in LLMs, illustrated by pelicans on b... by MichaelZuo on Sunday June 08, 2025 @09:26AM (#44216867)
I imagine the straightforward reason is that the “better” models are in fact significantly smarter in some tangible way, somehow.
Re:The last six months in LLMs, illustrated by pelicans on b... by more-nitor on Sunday June 08, 2025 @09:45AM (#44216968)
I just don't get the fuss from the pro-LLM people who don't want anyone to shame their LLMs...

people expect LLMs to say "correct" stuff on the first attempt, not 10000 attempts.

Yet, these people are perfectly OK with cherry-picked success stories on youtube + advertisements, while being extremely vehement about this simple experiment...

...well maybe these people rode the LLM hype-train too early, and are desperate to defend LLMs lest their investment go poof?

obligatory hype-graph classic: https://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Ga...

Re:The last six months in LLMs, illustrated by pelicans on b... by og_kalu on Sunday June 08, 2025 @03:02PM (#44218895)