News for hackers. Stuff that matters.
Re:Hot take: GPT 4.5 is a nothing burger by echoangle on Saturday March 01, 2025 @05:50PM (#43224723)
I actually don’t, can you explain it?

Doesn’t the AI model basically take an input of context tokens and return a list of probabilities for the next token, which is then chosen randomly, weighted by the probability? Isn’t that exactly the definition of a Markov chain?

Re:Hot take: GPT 4.5 is a nothing burger by fooker on Saturday March 01, 2025 @07:08PM (#43225587)
You have almost answered the question.

LLMs basically return a Markov chain every single time. Think of it as a function returning a value vs returning a function.

Now, I'm sure a sufficiently large Markov chain can simulate an LLM but the exponentials involved here would make the number of atoms in the universe a small number. The mechanism that compresses this down into a manageable size is famously 'attention is a you need.!'

Re:Hot take: GPT 4.5 is a nothing burger by JohnKemeny on Sunday March 02, 2025 @06:18AM (#43229314)
> Now, I'm sure a sufficiently large Markov chain can simulate an LLM but the exponentials involved here would make the number of atoms in the universe a small number.

No, LLMs are a Markov chain. Our brain, and other mammalian brains, have feedback, strange loops, that a Markov chain doesn't. In order to reach reasoning, we need to have some loops. In that way, RNNs where much more on the right track towards achieving intelligence than the current architecture.

Re:Hot take: GPT 4.5 is a nothing burger by fooker on Sunday March 02, 2025 @07:34PM (#43236956)
A modern reasoning model is exactly a feedback loop.
Re:Hot take: GPT 4.5 is a nothing burger by seanhunter on Sunday March 02, 2025 @10:53AM (#43231653)
They have the Markov property that the next state depends only on the current state (context window plus the weights of the model) do they not? Any stochastic process which possesses that property is a Markov process/chain I believe.
Re:Hot take: GPT 4.5 is a nothing burger by fooker on Sunday March 02, 2025 @07:35PM (#43236972)
The universe has that property too.

But no, most LLMs have a tweakae 'temperature' parameter that introduces some randomness and sometimes have very interesting results.

Re:Hot take: GPT 4.5 is a nothing burger by seanhunter on Sunday March 02, 2025 @11:36PM (#43238346)
Temperature just changes how an llm samples from the token distribution so I wouldn’t think that constitutes “memory” that would contradict the Markov property