Re:The last six months in LLMs, illustrated by pelicans on b...
by diggan on Sunday June 08, 2025 @09:39AM
(#44216939)
Yeah, this is the problem with benchmarks where the questions/problems are public. They're valuable for some months, until it bleeds into the training set. I'm certain a lot of the "improvements" we're seeing are just benchmarks leaking into the training set.
[ Parent
| on HN ]
Re:The last six months in LLMs, illustrated by pelicans on b...
by travisgriggs on Sunday June 08, 2025 @10:52AM
(#44217361)
That’s ok, once bicycle “riding” pelicans become normative, we can ask it for images of pelicans humping bicycles.
The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible. A plausibility machine (LLM) will struggle with the implausible, until it can abstract well.
Re:The last six months in LLMs, illustrated by pelicans on b...
by diggan on Sunday June 08, 2025 @11:54AM
(#44217725)
> The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible
Until there is enough unique/new subject-verb-objects examples/benchmarks so the trained model actually generalized it just like you did. (Public) Benchmarks needs to constantly evolve, otherwise they stop being useful.
Re:The last six months in LLMs, illustrated by pelicans on b...
by demosthanos on Sunday June 08, 2025 @12:57PM
(#44218087)
To be fair, once it does generalize the pattern then the benchmark is actually measuring something useful for deciding if the model will be able to product a subject-verb-object SVG.
Re:The last six months in LLMs, illustrated by pelicans on b...
by zahlman on Sunday June 08, 2025 @01:10PM
(#44218176)
I can't fathom this working, simply because building a model that relates the word "ride" to "hump" seems like something that would be orders of magnitude easier for an LLM than visualizing the result of SVG rendering.