News for hackers. Stuff that matters.
Re:Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearl... by belter on Saturday April 05, 2025 @02:17PM (#43595416)
Actually yes...If the kid spends their whole life in the box and never invents a new block, that’s just combinatorics. We don’t call a chess engine ‘creative’ for finding novel moves, because we understand the rules. LLMs have rules too, they’re called weights.

I want LLMs to create, but so far, every creative output I’ve seen is just a clever remix of training data. The most advanced models still fail a simple test: Restrict the domain, for example, "invent a cookie recipe with no flour, sugar, or eggs" or "name a company without using real words". Suddenly, their creativity collapses into either, nonsense (violating constraints), or trivial recombination, ChocoNutBake instead of NutellaCookie.

If LLMs could actually create, we’d see emergent novelty, outputs that couldn’t exist in the training data. Instead, we get constrained interpolation.

Happy to be proven wrong. Would like to see examples where an LLM output is impossible to map back to its training data.

Re:Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearl... by fragmede on Saturday April 05, 2025 @02:22PM (#43595466)
The combinatorics on choosing 500 pieces (words) out of a bag of 1.8 million pieces (approx parameters per layer for GPT-3) with replacement, and order matters works out to be something like 10^4600. Maybe you can't call that creativity, but you've got to admit that's a pretty big number.
Re:Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearl... by belter on Saturday April 05, 2025 @02:29PM (#43595543)
I said No handwaving with scale. :-)
Re:Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearl... by fragmede on Saturday April 05, 2025 @02:55PM (#43595861)
Right—but why should “new ABS plastic” be the bar for creativity? If the kid builds a structure no one’s ever imagined, from an unimaginably large box of Lego, isn’t that still novel? Sure, it’s made from known parts—but so is language. So are most human ideas.

The demand for outputs that are provably untraceable to training data feels like asking for magic, not creativity. Even Gödel didn’t require “never seen before atoms” to demonstrate emergence.