Re:The last six months in LLMs, illustrated by pelicans on b...
by demosthanos on Sunday June 08, 2025 @12:57PM
(#44218087)
To be fair, once it does generalize the pattern then the benchmark is actually measuring something useful for deciding if the model will be able to product a subject-verb-object SVG.
Until there is enough unique/new subject-verb-objects examples/benchmarks so the trained model actually generalized it just like you did. (Public) Benchmarks needs to constantly evolve, otherwise they stop being useful.