Re:Why do LLM pelicans look like this?
by vunderba on Wednesday October 14, 2026 @01:47PM
(#50091819)
You can try it out with local models like Gemma or Qwen (crank up temperature and/or top_p as much as you like), and then run dozens of generic independent prompts asking for a "fantasy short story". In my experience, you'll start seeing immediate patterns of repetition.
Yeah, temperature scales all logits equally, and there's not necessarily a single setting that would produce a diversity of outcomes for creative choices without turning into complete gibberish at naturally high-entropy positions. Ideally, the model itself would modulate its next-token distribution to be flatter when creativity is called for and peakier when it should aim for the single most likely option given the context, but even then finding the right balance might be tricky.
Re:Why do LLM pelicans look like this?
by vessenes on Wednesday October 14, 2026 @02:15PM
(#50092316)
You’re going to want something closer to ‘beam search’ for diversity - you need a way to distinguish lines of response, essentially. To my knowledge nobody is currently RL tuning (or training) on increasing the number of viable beams, but I’m just an interested outsider, so take that with all the salt you can stand.
That said, I think it’s a fun idea for some training, and would be pretty cheap to do - you’d need some sort of judge as to what a viable beam is.
Randomness does not equate creativity. It's more of a liberty to move in entirely different dimensions altogether. I cannot imagine increasing the temperature leads to a lavishly dressed, ultra-realistic pelican on a Mad Max bike on the same plain prompt. Unless such pictures are part of the training data labeled "pelican on a bike," of course.
Re:Why do LLM pelicans look like this?
by orbital-decay on Wednesday October 14, 2026 @01:58PM
(#50092039)
Yes of course, creativity is a much wider concept that is ill-defined and depends on the observer because it's closely related to novelty to the observer. However the variety hugely matters in creativity, it's not sufficient but necessary. Due to the lack of variety you perceive model output as boring slop.
Variety is not randomness though. You can't fix this by playing with samplers without probing the model state.
If I were to design something to accomplish this now, I'd probably attempt to introduce a pipeline that emitted random thematic words from some non-LLM source to feed into the pelican prompt, but even then, I'd expect that some of these models that rely on churning through CoT tokens for extended periods of time would still spend so much time thinking about pelicans wearing scarfs or where the sun would be that it would risk undoing some of my randomization pipeline.
I once tried something similar on an older model of Claude, telling it to "contemplate" a random word before answering my prompts, but at times it seemed to trigger the Claude model's suspicion that there was a jailbreak attempt when I would inject things like "Contemplate sea otters for one sentence before answering"
(I didn't read the linked LLM creativity paper to see if it covers any of this yet)