Re:Hot take: GPT 4.5 is a nothing burger
by JohnKemeny on Sunday March 02, 2025 @06:18AM
(#43229314)
> Now, I'm sure a sufficiently large Markov chain can simulate an LLM but the exponentials involved here would make the number of atoms in the universe a small number.
No, LLMs are a Markov chain. Our brain, and other mammalian brains, have feedback, strange loops, that a Markov chain doesn't. In order to reach reasoning, we need to have some loops. In that way, RNNs where much more on the right track towards achieving intelligence than the current architecture.
A modern reasoning model is exactly a feedback loop.
Re:Hot take: GPT 4.5 is a nothing burger
by seanhunter on Sunday March 02, 2025 @10:53AM
(#43231653)
They have the Markov property that the next state depends only on the current state (context window plus the weights of the model) do they not? Any stochastic process which possesses that property is a Markov process/chain I believe.
The universe has that property too.
But no, most LLMs have a tweakae 'temperature' parameter that introduces some randomness and sometimes have very interesting results.
Re:Hot take: GPT 4.5 is a nothing burger
by seanhunter on Sunday March 02, 2025 @11:36PM
(#43238346)
Temperature just changes how an llm samples from the token distribution so I wouldn’t think that constitutes “memory” that would contradict the Markov property
LLMs basically return a Markov chain every single time. Think of it as a function returning a value vs returning a function.
Now, I'm sure a sufficiently large Markov chain can simulate an LLM but the exponentials involved here would make the number of atoms in the universe a small number. The mechanism that compresses this down into a manageable size is famously 'attention is a you need.!'